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"He that publishes J book runs a great hazard," said^ervantes four centuries ago. 
"Would that my enemy would write a book," said another; probably a politician 
whose name is obscured by time. 

Such aphorisms wfere once highly respected. But people are now writing books as 
fast as publishers/will print them. 

This prime study group was charged with the responsibility of developing "A Train- 
ing Guide for Evaluating Vocational Rehabilitation Programs and Services. With 
this in mind, we have endeavored to produce a document that can be utilized by 
state vocational rehabilitation agencies in developing a system for evaluating either 
their overall program or specific aspects of it. 

Therefore, at the risk of running a great hazard and/or subjecting ourselves to our 
enemies, we have taken on this important challenge. The document which follows 
represents the collected efforts of many people who devoted considerable time and 
energy to present a guideline which will be useful to those interested in developing 
a sound evaluation program in their agency. 

We are all aware of the need for and the importance of evaluating performance. 
Evaluation can serve many purposes. It can help us understand what we are doing 
or not doing. It can help us bring about change or it can help us strengthen our ■ 
present position. Needless to say , evaluation is a must if we are to properly fulfill 
our role in vocational rehabilitation of assisting the handicapped in finding their 
rightful place in society. 

With these things and many others in mind, we have prepared the following 
document. It is, as its title states, a guide for further study, investigation, and 
development. 
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CHAPTER I 



PROGRAM EVALUATION: WHY DO IT? 




Why do program evaluation? 

Why is it important? 

We live in an age of accountability. 
It has been said that the decade of 
the 60's was the decade of expan- 
sion and innovation in vocational 
rehabilitation, but that the decade 
of the 70's is the decade of account- 
ability. Schools, businesses, and 
public-serving agencies have, in 
recent years, been put under more 
and more pressure to prove that 
whtat they're doing is worthwhile. 



Rehabilitation is no different. 



The day when legislatures were satisfied at appropriation time with figures on the 
numbers of clients served is fast disappearing. If you were asked to testify before 
a legislative appropriation subcommittee and were asked the question, "How well 
do you serve the clients you serve?" could you answer with any degree of accuracy? 
Think about it. 

In addition, rehabilitation is rapidly moving into serving entirely new populations, 
with varied and diverse kinds of programs. Public Assistance recipients, drug addicts, 
alcoholics, are now within the purview of rehabilitation activities. Do all these pro- 
grams work? How well do they work? What have we learned from trying to serve 
them? Can you answer these questions? Think about it. * t 

Rehabilitation certainly is growing; all these new programs with different kinds of 
clients. But are there populations not served right now that need to be? We're al- 
most in complete agreement that there are many needs not presently being served 
by rehabilitation. But what are they? How can we find out? Once discovered, 
how can we meet the needs? These are pressing questions that need to be answered. 
Think about it. 



Federal and State administrators are constantly demanding accountability of us in' 
terms of program functioning. For example, when we work with Public Assistance 
clients we must work toward decreasing dependency. How well are we doing? 
Our boss must know; he's accountable to somebody else, and he needs the answers 
too. Can we answer these kinds of very natural and understandable questions 
with any degree of accuracy? Think about it. 

All the above questions, when addressed, have tremendous ramifications in terms 
of organizational development. Each and every one, for example, has a bearing on 
staff training. What kinds? How long? Who should be trained? Think about it . 

These considerations, coupled with legal basis and authority established by the 
Amendments to the Vocational Rehabilitation Act, logically require a systematic 
plan for program evaluation to help the agency: 

(1) To determine agency goals. 

(2) To determine the effectiveness of agency service delivery. 

(3) To better determine vocational rehabilitation program needs. 

(4) To insure efficiency in program management. 

Determining apency goals 

Growth and expansion always create the necessity for the reassessment of goals 
and objectives. Service priorities continually shift with respect to changing eco- 
nomic patterns, geographic factors, and special disability groups. In order to more 
effectively plan and devel6p a service plan which addresses itself to these external 
changes, emphasis and practices must be systematically reevaluated. 

It has been said that, "No data exists to show that vocational rehabilitation ser- 
vices are not dissipated on clients who could achieve employment goals without 
the high quality, expensive services offered." Whether this statement be true or 
not, its meaningfulness can only be interpreted in light of established program 
objectives. Is it a pix>gram objective to serve only cliefits who could not achieve 
employment goals without the "high quality , expensive services offered"? We can 
only determine effectiveness of service delivery by evaluating objectives. 

Determining the effectiveness of agency service defivery 

Vocational rehabilitation agencies exist for the purpose of delivering meaningful 
vocational rehabilitation services to disabled persons. Some operations researchers 
feel that specific factors exist that contribute to client success and failure. Indeed, 
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if this be true, a method of effective program evaluation could serve to uncover 
strengths and weaknesses,either internal or external, that may be corrected or 
circumvented. 

Perhaps one of the greatest expenditures which confronts vocational rehabilitation 
agencies is the purchase of training services from vendors. It is important that 
these services be evaluated to determine how adequately they meet client employ- 
ment needs. 

Vocational rehabilitation services vary according to area and regional differences. 
It is reasonable to assume that the Blade Lung disease is more pronounced in coal 
mining areas than in the coastal plain. Evaluation in order to establish local, area- 
wide, or regional priority levels is then necessary, if vocational rehabilitation 
agencies are to effectively address themselves to peculiar papulation needs. 

Determining vocational rehabilitation program needs 

/ 

The location ana identification of servite g3ps is one of the greatest missions of an 
effective evaluation unit. The identification of such gaps serves to determine over- 
all staff training needs, necessary program changes, and the justification or knowl- 
edge of necessary personnel needs. 

The assessment and evaluation of a local community's need, ability, and willing- 
ness to plan for and develop community resources may often be th$ key to facili- 
tating agency services. Evaluation units can be valuable assets in identifying areas 
of services that could be complemented by such resources. 

Insuring efficiency in program management 

Efficient management techniques must be practiced if the vocational rehabilitation 
pi%gram is to assure itself that the most people possible receive the best quality 
service within given financial means. Effective management practices must be a 
goal of all service agencies if efficiency is to be realized. Costs, organizational pat- 
terns that provide for expedient services, and personnel assignments can be critical 
factors in providing effective services to the disabled. Standardized accounting pro- 
cedures may vary little from agency to agency, but cost-analysis can present im- 
portant information for decision makers. A systematic evaluation of agency man- 
agement and fiscal practices not only serves to enhance public confidence, but 
allows program administrators the opportunity to realistically determine the level 
of efficiency with which their program operates. 
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Levels of involvement 



l/i a typical rehabilitation service system, several levels can be distinguished: 

(1) Social and Rehabilitation Service (SRS) - concerned with alterna- 
tive social service strategies, allocation arnong rehabilitation and 
other programs and needs. 

(2) Rehabilitation Services Administration (RSA) - concerned with 
broad policy goals, and needs of total state programs, 

(3) State Vocational Rehabilitation Agencies - concerned with delivery 

of services in the aggregate to meet the needs of clients and the de- 
sire of the community for rehabilitation; obtaining more slate funds 
and matching funds from the Federal government and using them 
well; and planning and evaluation. 

(a) Regions - concerned with balancing money and per- 
sonnel resources among functional subdivisions (dis- 
tricts). 

(b) Districts - concerned with actual service delivery to 
clients in localized areas, 

(c) Supervisors - concerned with control of quality and 
quantity of services delivered by counselors, 

(d) Counselors - concerned with needs of clients, require- t 
ments of superiors; directly responsible for casework. 

(e) Clients ■ final recipients of services; contact with 
system usually confined to district office and below. 

In descending the levels in the system, from SRS to clients, concerns become less 
and less global, and more and more microscopic. The types of evaluation and the 
criteria selected might also vary at the different levels where evaluation might take 
place. It is the responsibility of the evaluator to determine precisely whom he is 
to serve, and what types of evaluation might apply to that level in the hierarchy. 

Cutting across these levels are other programs. These additional perspectives 
should be noted, although evaluation of these programs is done according to state 
agency objectives. These are: 

(1) Other public agencies - such as those represented by rehabilitation 
programs whieh serve welfare recipients, trust fund recipients or * 
other specially funded projects, 

(2) Public governing bodies sponsoring jointly funded rehabilitation pro- 
jects>such as a county school district or a county alcoholism clinic. 

(3) Private community agencies or vendors of rehabilitation services, such 
as workshops. 
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The next group of items are specific problems which have been identified in reha- 
bilitation programs as requiring evaluation and analysis The list could readily be 
extended. We have included it to stimulate further discussion by state agency 
administrators and staff of the needs for evaluation. What other problems would 
you add to the list as needing attention? What problems should receive priority? 

SUMMARY 



Why Dolt? 

(1 ) To assess and evaluate the agency's current situation and set future goals and 
priorities. 

(2) To locate gaps in service. 

(3) To evaluate counselor and other staff performances. 

(4) To determine future objectives. 

(5) For cost analysis. 

(6) To make better staff assignments. 

(7) To determine and evaluate training needs. * 

(8) To determine if established and/or projected goals are being 'achieved. 

(9) To justify quality of agency performance. 

(10) For public relations valuesjustifying budgets, etc. 

(1 1) To modify and/or improve existing organizational structure. 

(12) To determine and justify personnel and budgetary needs. 
713) To determine facility and other program needs, 
pt) To eliminate weak and/or unnecessary program activities. 
■ 5) To evaluate and-strengthen relationships with other rehabilitation and vendor 
^ facilities. 

(1 6) To isolate success and failure factors in client service programs. 

(17) To isolate and evaluate service programs for priority target populations. 

(18) To evaluate personnel practices in terms of recruitment and job assignments. 

(19) To evaluate the quality of service by vendors. 

(20) To justify the elimination or continued # use of various categories of vendors. 

(21 ) To evaluate the implementation of Title VI requirements as an integral part of 
quality case service (Civil Rights Act). 

(22) Vo determine the neecf for changing and/or improving training programs and/or 
providing additional training opportunities. 

(23) To isolate positive and negative factors in agency operations. 

(24) To provide for greater staff input in establishing agency goals and objectives. 
(2£) To provide for client and other citizen participation and input. 



' Types of questions that invite evaluation 



(1) Is the current general mix of rehabilitation services under the Federal-State 
program, in sum, helping the disabled person? Does he actually benefit from 
this help, or is he likely4o be just as successfully rehabilitated if he never 
gets to the Vocational Rehabilitation program? What are the personal and 
program elements responsible for these outcomes? 

(2) Within the cunent general mix of rehabilitation services, which services or 
which mix seems to be the critical one in a successful rehabilitation - gener- 
ally, by disability group, and in interaction with other relevant client charac- 
teristics? 

(3) What has happened. in Vocational Rehabilitation's rare efforts to move towards 
decentralization of State programs? What are the factors that led the few 
efforts to abort? Why has Vocational Rehabilitation been slow to move in 
this direction? 

(4) What has happened to Vocational Rehabilitation's efforts to take its individ- 
ualized service model and adapt it to other groups? 

(5) What has happened under the extended evaluation authority? Have its use 
and results suggested the need for altering our basic authority under section 
2 or the need for altering the extended evaluation authority? 

How effectively are the State programs being administered in terms of : 

(a) administration (including planning, operations, and financial 
management), • 

(b) eligibility determination, and 

(c) choice of vocational objectives for the disabled (including 
level and stability of employment)? 

(7) How effective and balanced is the Vocational Rehabilitation program in its 
efforts to rehabilitate the various groups of disabled who are receiving public 
support; e.g.. public assistance recipients, social security disability recipients, 
and institutional populations of the mentally ill and mentally retarded? 

(8) Is Vocational Rehabilitation making increased effort to rehabilitate persons 
with progressive disabilities such as multiple sclerosis? Have we restricted 
our approach or are we ready to move more actively into these disability 
groups? What are the relevant factors to be taken into account in deciding 
to move ahead? 



(9) What stretch of interpretation of behavioral disorder as a mental disability 
is necessary in order to make the disadvantaged eligible for Vocational 
Rehabilitation? What have we done or learned from past efforts in this 
regard? 

(10) What success has Vocational Rehabilitation had in the last decade with selec- 
tive individualized placement of the disabled in; 

(a) self-employment (including homebound), 

(b) sheltered employment, and 

(c) small business enterprise? 

(1 1) What kinds of disabled people are the Vocational Rehabilitation programs: 

(a) rejecting from referral, and 
, (b) closing as not rehabilitated? 

(12) Why must the Vocational Rehabilitation program serve three persons for 
every rehabilitation? Could this proportion be reduced? 

(13) What are the factors that keep Vocational Rehabilitation from working 
effectively at the local level with the public assistance and employment 
services? 

(14) What are the special problems of the aged mentally retarded? What special- 
ized -services are needed by this group? What should be the role of generic 
agencies in providing services to the aged mentally retarded? 

(15) How are mental retardation facilities utilized? How can they be better util- 
ized? How is space used in off-hours? What is the best location for mental 
retardation facilities? 

(1 6) What are optional staffing patterns for mental retardation facilities? How 
do staffing patterns relate to community needs, cost, and other variables? 

(1 7) What kinds of simple otyective methods can be developed to assist staff to 
make self -evaluation directed toward constructive remedial action? 

(18) How can research utilization in the field of mental retardation be improved? 
How can dissemination of information on mental retardation program needs 

N^// be improved? 
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(19) What are the effects of early intervention on programs for the mentally 
retarded in terms of reduced costs and utilization of other resources for 
vocational training in later life? 

(20) What is the feasibility of moving toward program planning within a State 
utilizing designated geographic service planning units to determine Voca- 
tional Rehabilitation service needs and the availability of public and private 
Vocational Rehabilitation sector resources to meet those needs? 

(21 ) What should be the role of advocacy for Vocational Rehabilitation in arrang- 
ing for and/or providing, when necessary, rehabilitation services for handi- 
capped individuals as distinguished from Vocational Rehabilitation services 
for disabled? 

(22) How can utilization of rehabilitation facilities be improved and increased? 

(23) What changes in staffing pattern^ and programming sire necessary for work- 
shops to improve services for the disadvantaged? 

(24) Within states, what are the patterns of service delivery penetration by geo- 
graphical areas, and what area characteristics are related to high or low pene- 
tration? 

(25) In states which are effective in terms of rehabilitation per 100,000 popula- 
tion, per counselor, cost per rehabilitation, etc., what is the quality of ser- 
vice provided? Also, what administrative and procedural characteristics 
prevail? 

(26) To most efficiently allocate case service funds for states, what should be 
the mix of disabilities treated? 

(27) What actually is the role of the rehabilitation counselor? Is there a more 
appropriate role to be played? 

(28) How can workshops increase the flow of sub-contracts which are their main 
source of income? 

(29) Is it possible to develop quantifiable measures ofeffectivencss and appropri- 
ate scales for gauging progress toward "Management by Objective" goals? 
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CHAPTER II 
PROGRAM EVALUATION: ITS NATURE 




Finding out whether or not things work, and how to make things work better - 
that's what program evaluation is all about. 



The activity of finding out whether or not things work is the point of departure 
for the program evaluation process. But, it is an activity which is poorly under- 
stood. 



Much of this chapter will focus on explaining and justifying this activity. 

This activity is not the sole charge of the evaluator, hovyever. The program evalu- ^ 
ation process and, thus, the evaluator must intrude upon many other activities 
and function? which arte critical to the life of a rehabilitation agency. Setting goals 
and targets, planning, program management, monitoring and review, policy and 
program reassessment, executive decision-making, and decision-implementation. 
The evaluator does not have exclusive authority or even prime responsibility for 
any of these functions. Rather, the role of the evaluatorfc to act as a stimulator, 
as a catalyst, as a change agent in the organization, helpftrig to see that each of the 
above functions are carried out in such a way as to improve, the effectiveness of 
the program. The program evaluator will often find himself acting not only as an 
evaluator, but also as a participant in program planning and even policy formica- 
tion. As the evaluator plays these many roles, he must be careful to keep each role 
clearly distinct in his own mind and in the perception of others.' 
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Let us first focus on the activity of finding out. This finding out business 
can get pretty sticky.^ 

Enter Perry Mason - or the problem of bias 

The main gist to program evaluation is developing and sticking with certain 
rules of evidence. Rules of evidence is a term from the legal profession. There 
are all kinds of rules about what prosecution and defense can do in presenting 
evidence. Every time you hear Perry Mason up and say "Objection, your 
Honor" he's probably claiming that one of the rules of evidence is being 
violated. So his client isn't getting a fair shake. Of course, it's up to the 
judge to decide whether Perry's claim is justified or not. The judge enforces 
the rules of evidence. He is there to make sure the game is played fair and 
square - that bias is kept out of the proceedings. 

The nature of program evaluation is little more than a collection of rules of 
evidence that allow you to conduct evaluation with the assurance that bias is 
kept out — including your own. 

A detour - evaluation as an organizational function 

But before we go ahead, there are a couple of things we'd like to bring out. 




Now it^might seem like a detour. But if 
you bear with us, we think you'll see what 
we're driving at." 



We have two basic points to make before 
g<5ing into specifics: 



(1 ) Program evaluation is an 



aspect of the Agency's total 
organizational functioning. 



Program evaluation is an 



, organizational functio n* 
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(2) The extent to which program evaluation contributes to an Agency 
is influenced by how effectively the Agency functions as a total 
organization. Organisational function ing sets limits to program 

Organizational effectiveness - what's that? John Gardner (October 1965 issue of 
Harper's) wtote an article titled, "How To Prevent Organizational Dry Rot." 
Gardner answers the question of what constitutes an effective organization. 
Gardner maintains "an effective organization is a self-renewing one." 

How effective is your agency? 

Like to know whether your organization is self-renewing or not? Gardner has 
five questions you can ask to find out. 

x Ask yourself: 

( 1 ) How effective is my Agency's program for recruiting 
and developing talent? 

(2) How good an environment does my Agency provide 
for the individual e mployed ? 

(3) Does my Agency have built-in provisions for self- 
criticism? 

(4) How flexible is the internal structure of my Agency? 

(5) Does my Agency have means to combat the process 

by which people become pri soners o f their procedures? 

Them as has sets 

Go back to that five-item quiz a minute. Look at questions 3 and 5. Do you see 
how program evaluation relates directly to these two issues? ■ Program evaluation 
is both the built-in provision for self-criticism and a means by which organizations 
: get out of being jailed by their procedures. 

But there is something even more fundamental here. And, it's a little like a chicken 
and egg situation. Namely. self-renewmg agencies are going to be the very ones most 
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likely to develop and actively support program evaluation. Conversely, agencies 
that are loath to self-criticism are most likely to be the ones to accept program 
^tkluation results only when these results confirm existing sacred cows. 

Agencies in need of program evaluation most get it least. As we said, it's like a 
chicken and egg situation. Unless you're innovation prone and open to self- 
criticism, you're not going to give anything other than lip service to program 
evaluation. 

Them as has, gets! 

Testing your agency's SAPE S 



(2) What percent of the budget is spent on program evaluation? I made 
a distinction between what's budgeted and spent . Reason? Say you 
have a program evaluation unit with a budget of $60,000. But if the 
people with the program evaluation label are alfco the guys with a 
mess of other tasks (e.g., public relations, fiscal), then there's a whale 
of a difference between what an agency looks like it's spending and 
what is actually spent for program evaluation. 

( 3 ) How high u p in my A gency's hiera rchy is the program evaluation 
operation? As a rough rule of thumb, the higher an activity is 
located in the hierarchy, the more priority an agency gives to that 
activity. If Agency A has an Assistant State Director's position 
completely devoted to program evaluation, it suggests that Agency 
is more serious about program evaluation than Agency B which has 
program evaluation nestled off in fiscal or personnel. 




We just said that everyone more or less 
has to at least come out in favor of 
program evaluation. SAPE stands for 



Serious About Program Evaluation. 
If you want to test your Agency's 
SAPE, ask yourself questions like: 



(1) 



What percent of mv Agency's 
budget is earmarked for 
program evaluation? 
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Another thing - everything else being equal - isn't the Assistant Director more likely 
to get the ear of the State Director (and get him to pay attention to program evalua- 
tion results) than someone much further down the organizational ladder? So there's 
an additional reason for associating position in the hierarchy with SAPE. 

(4) Is the program evaluation unit a valued (or desirable) opportunity 
within my Agency? Every organization - has to develop places 
where they can put people who have flubbed up yi other more criti- 
cal areas. If program evaluation is that type of unit within an agency, 
it suggests less SAPE. Why? Simple. It, by definition, is a "less 
critical area" than the units the guy flubbed up in. Also, word soon 
gets around that a "promotion" to program evaluation is a promo- 
tion into a dead-end street. Result? Your bright up-and-coming - 
talent steers clear of program evaluation. This in turn leads to less 
competency in the program evaluation unit. A more tarnished image 
for program evaluation. And the snowball continues. 

Turning the corner 

Having made our detour, we'd now like to turn 
the corner and head back to the program evalua- 
tion process. We hope now that you see how things 
like an agency's capacity for self-renewal and its 
actual commitment to evaluation must be examined 
when you look at the nature of program evaluation. 
We hope, too, you see how program evaluation is 
an organizational activity and must lean on other 
organizational activities to be successful. 

In short, even if your program evaluation unit is a top-notch one; even if evaluation 
efforts are as valid as they can be; when it's all said and done'program evaluation is 
as good as your agency lets it be. 

Steps in evaluation process 

... — jf> ' . 
We are now ready to get down to considering the nature of program evaluation in 
more detailed terms. Like we said at the outset, the nature of program evaluation is 
a little more than the following of a series of rules - rules of evidence. Adhering 
tp these rules increases the chances that bias is kept out of the evaluation process. 
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Those who write about program evaluation (Hawkridge) icier to it as a logical, step- 
by-step process. These steps include. 

(1) Spell -out objectives for the program being evaluated. 

(2) Select objectives to be evaluated and determine statis- 
tical analysis. 

(3") Construct (or select) evaluation instruments. 

(4) Select samples. 

(5) Determine the points at which testing is to take place. 

(6) Conduct statistical analysis. 

(7) Develop conclusions and recommendations. 

In the pages that follow, steps 1 through 4 will be treated individimlly : Jteps5__ 
through 7 are condensed in a discussion of the framework for prog^me^aluat.on. 

S pell out program objectives 

You might argue whether or not spelling out program objectives is really part of 
program evaluation. Often program evaluation can't proceed because the objec- 
tives of the program were never really spelled out in very clear terms. It's a little 
bit like going on a trip with no destination. If you don't have a destination, it s 
hard to know when you've arrived. 

When we say "spell out." we mean that any program (or sub-program) needs to 
have an explicit statement in performance terms as to what that program is de- 
signed to accomplish. The key word in the sentence you just read is "performance 
terms." We distinguish program objectives in performance terms frdm program 
objectives couched m non-performance terms. 

For example, the statement that a program goal is to rehabilitate the handicapped 
is a non-performance statement. Among the reasons that it is not a performance 
statement is that it contains no indication of what constitutes rehabilitation, nor 
what is meant by the handicapped. Also it lacks a quantitative measure. There 
are many other reasons why a statement such as the one we have just used is not 
a performance statement. In other words, even if we were to s F cify what we 
meant by rehabilitation (e g.. one month continuous employment) and even if 
we spelled out eligibility requirements, we still would not have a satisfactory state- 
ment of program goals in performance terms. 

We hope you arc beginning to get the feel ol what we are driving at When we use 
the term performance. Performance simply means what concrete, objective. 
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observable t*aOs would have to be present for someone to conclude that program 
success has been achieved. 

A good performance statement of program goals should really have two ingredients 
in it. First, it should have a statement as to the extent to which program goals 
must be achieved. Secondly, a statement of the limiting conditions (if any) that 
would qualify the evaluation of whether or not performance goals have been 
achieved. 

V 

Extent f 

The notion of extent usually carries with it a quantitative idea. For example, to 
improve on our statement of program goals, we might add a statement to the effect 
that the goal of Program X is to achieve continuous employment for one month 
with one thousand psy chiatrically disabled clients. Maybe you can get a feel as to 
the increased precision that has brought about frwh the adding of this quantitative 
anticipation regarding the extent of program gbals. Qualifications regarding extent 
can be carried on much more beyond the little example we have just used. Geo- 
graphic distribution (e.g., state-wideness) is a commonly applied yardstick^Wving 
to do with the extent of program services and goals. 

Limiting conditions 

So far we've only talked about extent. We haven't said anything about the notion 
of limiting conditions. Limiting conditions, as the term suggests, has to do with 
any significant qualifications whose presence or absrnce is expected to have a 
marked effect on program evaluation. 

As an example of limiting conditions (and continuing with the example we've 
been using), let's talk about prior employment history. It is widely recognized 
that the chances of rehabilitation go up if - prior to disablement - an individual 
has had a stable work history. The same holds true for psychiatric patients. So 
in the statement of goals regarding (say) a special project designed to rehabilitate 
psychiatric patients, a statement regarding the allowable number of patients (so 
to speak) with good versus poor pre-morbid work history might be highly relevant 
to evaluating program effectiveness. 

Another example of a limiting condition might he the nature of the employment 
activity into which a client is closed. Most rehabilitation practitioners would con- 
cede that the rehabilitation of one thousand males into competitive remunerative 
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employment would probably represent a fnore stringent criterion of program suc- 
cess than the rehabilitation of one thousand homemakers So a good statement 
of program objectives will include all relevant conditions that bear on evaluation 
of program outcomes. 

This sounds like a rather tedious task, doesn't it? The reason it sounds like it's a 
tedious task is that it is! The writing of program objectives in good, bolid perform- 
ance terms is a taxing job. It would not be surprising (for example) that a detailed 
statement of a state agency's performance goals for a fiscal year might consume 
several pages. 

As suggested before, you might question whether or not this is really a program 
evaluation function^ And if pressed, we might have to agree with you. But the 
idea is this. Program evaluation can only start after specified program goals have * 
been arrived at. And everything else being equal, program evaluation will be a 
lot easier when the goals of the program have been clearly set forth in concrete 
performance terms, But when we get into the business of goal-setting, lo and 
behold we find ourselves back into the whole business of management, adminis* 
tration and - you guessed it - organiz ational d evelopment. 

A well-developed organization is one with clearly defined goals - goals that have 
been arrived at as a result of participation by significant elements in the rehabili- 
tation community. 

In a real sense, then, program evaluation can't tell you what your goals should be. 
It can only tell you whether or not you've reached the goals you said you were 
going to reach. And it can only do this if you came to program evaluation with 
clearly stated performance goals. 

The example that comes into mind here is the statement of President Kennedy in 
1960 that a goal for the United States would be to have a man on the moon in 
ten years. Now you can debate whether or not that was a good goal or a bad 
goal* But it was a clearly- stated performance goal. And the nature of the goal 
was such that you didn't need a very high-powered evaluation effort to figure out 
whether or not the goal was achieved. Either an American wound up on the moon 
in ten years or he didn't. It .vas that simple. But the point is that the evaluation 
of whether or not the man-on-the-moon goal was achieved can only begin after 
the goal was explicitly set forth. 

Each participant in the rehabilitation process S R.S., R.S.A.. State Directors, 
fiscal chiefs, directors of field operations, (raining and staff development specialists 
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district administrators, supervisors, counselors, clients has needs, values and ob- 
jectives which may be somewhat unique and which may conflict. When values 
conflict there are likely to be problems in choosing criteria by which a program 
should be judged. The program may be judged desirable by one set of values and 
criteria but undesirable by another set of values and criteria. Or, value differences 
may result in different interpretation of a problem. For example, the counselor 
may see a client's m^jor problem as motivational while the client sees it as situa- 
tional. Or, administrators may ascribe program weaknesses to failings on the part 
of individuals, while others may interpret the same weaknesses as structural. 

Often objectives will be ambiguous such as to : " improve quality of life," 
"achieve vocational rehabilitation " "achieve self-sufficiency or maximum poten- 
tial." Objectives must then be sharpened and distinctions madfc between ulti- 
nate and immediate goals. The evaluator becomes the catalyst for prompting 
administrators or line staff to more explicit goals. In this process, many levels in 
the agency may participate in defining goals. This participation helps to assure 
understanding of goals and acceptance of their legitimacy. For example, the 
values of counselors and administrators may appear to conflict. Counselors often 
insist that they are concerned with "quality 0 While their administrators are con- 
cerned with "numbers." Yet, a better definition of goals by both might reveal 
that no conflict really existed. 

\ When evaluation is seen as a part of planning, the values upon which program 
^objectives are based are a consequer\pe, of the judgments arrived at through pro- 
gram evaluation. In most instances, however, values are formed and goals set 
before a program is to be evaluate! It is In this latter case that the evaluator works 
closely with operating-level staff in assessing the current situation to identify con- 
cerns to be evaluated and to determine objectives and criteria. Every effort should 
be made to recognize that differences in values exist and the values reflected in the 
final selection of criteria should be explicit. In this way the choice of issues is 
relevant rather than arbitrary and tie context within which criteria are selected is y 
well understood. V^/ 

So much for step one in the evaluation process. 

Select objectives to be evaluated and determine statistical analysis 

Let's assume now that you have gone ahead and developed a set of goals in clearly- 
defined performance terms. The next step is to select which of those goals you 
would like to measure. You minht say. "Well, why not measure all of them?" 
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In some cases, it might be feasibly>Jo evaluate program effectiveness for all of the 
goals set forth. In other uses, it might be financially or practically not feasible to 
evaluate all goals all at once (e.g.. limited program evaluation resources, limited 
time, etc.). 

Once again, it is not the function of program evaluation (strictly speaking) to 
determine which of the goals should be selected for evaluation. This, like the first 
step (spelling out objectives), is largely a question of policy and as such, it's a 
question of values, Now, just because a person is a program evaluator he should 
not be barred from having any opinions as to what program goals ought to be and 
what ones ought to be evaluated, Our only point is that when a person is function- 
ing in this way. he is not functioning as a program evaluator, he is functioning as 
a policy formulator A nd the processes of policy formulation must be always 
kept sharplv differentiated from issues of evaluation. ^ 

In selecting which performance objectives are going to be measured, an agency is 

likely to be influenced as much by external as by internal considerations. The 

development ofnational parities (e.g., the dibadvantaged) or the existence of 

peculiar local conditions (e.g., pneumoconiosis) might have as much to do with 

what aspects of program functioning will be evaluated as internal concerns and 
* 

considerations. 



Decisions about w,hat should be evaluated are also influenced by anticipations 
about the likejy outcome of the evaluation. For example, the consequences of 
a negative evaluation should be foreseen, p. rticularly if this could result in the 
termination of the program. The need is f rr a contingenc y analysis : i.e., the pro- 
cess of plotting out The most probable instances of "what would happen if . . /' 
Here, the judgment of the evaluator and the sponsor of the evaluation plays a 
significant role. 

The contingency analysis should include: 

( 1 ) an understanding of the entire program being evaluated 

(2) delineation of the place of evaluation in the program, 

The former suggests in what directions impacts might\e felt and where pro- 
gram modifications might be instituted. The latter demands a statement of the 
purpose of the evaluation. It may be either "formative." assessing progress to- 
ward an objective or "summative," assessment of final achievement, Of course, 
evaluation is hardly meaningful without an eye toward the realities of imple- 
mentation of findings, Possible program decisions, might include some of the 
following: 
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( 1 ) Expansion of the program 

(a) the program is working optimally and current agency expedience 
is anticipated as being a good predictor of what kinds of new 
clients will be served in an expansion. 

Termination of the program may be justified if- 

(a) the program's -goalsliave been achieved . ' 

(b) publiQ priorities determine that the program is of little merit 
relative to other felt needs. 

(c> the costs of operation endanger other valued programs, 
(d) no measurab>^gbservable effects of program operation are 
^ in evidence* 

(3) Continuation of the program (unchanged) may be based upon - 

(a) indications of success in njeeting objectives. 

(b) overwhelming public support for popular programs based on 
emotional or humanitarian reasons. 

(c) hope that improvements will eventually occur. 

(4) Revision of the program may occur at almost all points in the 
'-evaluation process - 

(a) new knowledge or capabilities might redefine the problem. 

(b) legislation might be modified to change the direction or 
emphasis. 

(c) a different use of resources (funds, facilities, personnel) 
might be proposed. 

(d) different management techniques could change the adminis- 
tration pf the program (Program Planning Budgeting 
System, Program Evaluation Review Technique). 

(e) changes in the staff might affect program outcomes (number 
of staff, educational level, experience, assignment of 

* responsibilities). 

(0 different client groups might be served, or the way in which 
. services are delivered might be changed. 

In each of the above cases>i*~is necessary to establish and apply ^ka^form of 
criteria to the program's operation. 
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Construct (or select) evaluation instruments 

The selection of evaluation instruments will be heavily influenced by whether the 
evaluation planning takes place before the fact or after the tact.^ 

Too much of evaluation in rehabilitation takes place after the fact. By that we" 
mean a special project or a new initiative of some type is undertaken and that 
little systematic thinking goes into evaluating the program in advance . No, it is 
much more common to have the program go on for some time and then to have 
questions raised regarding how well the program has been doing. This kind of 
hindsight evaluation has a couple of drawbacks associated with itwOne of the big- 
gest drawbacks is that of cost. Everything else being equal, it will cost more to 
evaluate adequately something that's already gone on rather than to plan evalua- 
tion from the outset. As simple a thing as the design of an efficient data collec- 
tion instrument can make a whale of a difference when it comes to evaluating 
theWults of a program. If we have to golo six different sources to collect the 
nece$sa?yv$ix pieces of information on a particular client, it's going to cost consid- 
erably more than if we had planned from the outset and designed a little form 
that brought the information together. 

Because of the increased costs of after-the-fact evaluation, it often happens that 
the validity of after-the-fact evaluation greatly suffers For this reason , agencies 
ought to develop evaluation strategies in advance of that's going to be evaluated 
rather thah after. In addition to decreasing cost and increasing validity, you can 
also accomplish a great deal more by going tins route. For one thing, you can 
start a dialogue between your evaluation , people and your service people in advance 
of the activity being evaluated. This will sometimes pay handsome dividends in 
terms of breaking dowo defensiveness and resistances between these two groups. 
If your service people have a chance to get together with your evaluation people 
before a project, they'll be mop^likely to see the evaluati6n in a less defensive 
light. 

Conversely, if your program evaluation people interact extensively with your ser- 
vice people, chances are the instruments and procedures they devise will be inore 
realistic than if they were operating without the benefit of such a review. , 

Sneaking of. instruments, an evaluation instrument can include everything from 
scorp^ on psychological tests to changes in annual salary. The evaluation instru- 
ment can be very expensive (e.g., an hour-long, individual interview) or compara- 
tively inexpensive (e.g., an itemfcn the R-300). It's beyond the scope of this 
presentation to go into which evaluation instrument is going to be the best in a 
particular evaluation situation. 
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What can be stated is this. If you select your evaluation instruments in advance 
of the program, the chances are you're going to ^get far more efficient and far more 
meaningful results from your evaluation efforts,*/ 

Select Samples 

J 

Sampling can get pretty confusing to people who haven't recently taken a re- 
search course. There are all kinds of words running around about samples. You'll 
hear words like random samples, representative samples/stratified samples, strati- 
fied random sample*, etc., etc. The basic concept behind a sample is really some- 
thing that we're familiar with from day-to-day experiences. We all, at one time 
or another, have probably had a blood test. A blood test is a perfect illustration 
of the sampling idea. 

Doctors take a tiny portion of our blood and - based on the analysis of that tiny 
portion -vthey are able to make .almost fool-proof statements about the condition 
of the endre bloodstream. It's very important to keep in mind that when the lab 
finishes its work on the tiny vial of blood', the doctor doesn't hesitate to make 
statements about the condition of our entire blood system. The reason the doctor 
can be so bold is die to the fact that he can assume that whatever is present in our 
bloodstream distributes itself in a random fashion. Because of this randomness, 
the doctor is iifafashion able to say - "You've seen one vial of his blood, you've 
seen it aH." 

The doctor, can make pretty Mrong conclusions based on a small amount of biood. 
So, too, if a large enough sample of clients is selected in a random fashion, then 
prefty definite conclusions can be drpwn - not just about the sample selected, but 
about the total group from which the sample came. It's difficult to over-emphasize 
the importance of randomness in the drawing of samples. .Because if you've got it, 
you're really in the driver's seat. By that is meant that the cost of doing evalua- > 
tions can be considerably reduced. Instead of examining folders on a thousand 
clients who have gone through a program, we can draw*a careful sample of one 
hundred of these clients and come up with results that are as certain as the results 
that would have been obtained from examining the Vhole thousand. While on the 
subject of samples, a few words should be said about control groups. 

yf 1 * 

Control Groups 

A control gnoup is also a sample. Contrp^oups are very seldom used in rehabili- 
tation research and there area couple pi reasons for it. For one tiling, if a control 
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group is to be a control group, it would have to be simitar to the serviced group 
in every respect (e.g.. age, sex; seventy of disability, etc.) except one. That one 
respect would be that the control group didn't get the rehabilitation program that 
the serviced group got. 

There is an understandable reluctance on the part of rehabilitation people to with - 
hold services. At times also there might even be legal complications associated 
with such withholding of services. 

But there are other times when objection to the withholding of services isn't 
really valid. Let's say someone has designed what he considers to be an improved 
work evaluation approach. But, he doesn't know whether or not it is in fact an 
improvement. To argue that this new technique should not be withheld from 
clients is to assume the very thing that ought to be proven Namely, is to assume 
that the new evaluation procedure is in fact an improvement. 

We could go on and on with the topic of sampling For the moment, it suffices 
to say that the addition of a control group to progi am evaluation enormously 
enhances the strength of positive findings. By positive findings, «; f S .ean findings 
in support of the value of the rehabilitation effort. Let us give an extreme ex- 
ample to illustrate our point. If we claim we have the new magic wonder Elixir 
that will cure your cold every single time, you'd probably be pretty interested, 
wouldn't you 0 Let's call this ole' G randpappv'S magic cough elixir . , Let us add 
that this elixir will absolutely cure your cold in two weeks from the time of appli- 
cation. Two weeks! Yes, you'd be perfectly right to reject this magic elixir since 
it really doesn't do more than what would have happened if you never had taken 
the elixir at all. 

What's our point? Regardless of how sincere you think your "particular magic 
elixir is, unless you have .some built-in way of showing that it is an improvement 
over what would have happened anyhow, any proof that yoy fcftETdn an improved 
product might be open to considerable question. So regar^ess of hov^ carefully 
we design a study and look at what the impact of the rehabilitation program is, 
our findings will always be a little short of being conclusive if we don't have a con- 
trol group present to find out what would have happened if we had never devised 
a particular rehabilitation program at all. 

Another way of looking at it is that control groups is a way of separating the 
Ole' Grandpappys from the Louie Pasteurs. 
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Now that we have seen an overview of the steps in an evaluation process, how do 
we actually go about evaluating an overall program? Whether we try to evaluate 
the overall rehabilitation program of a State agency, or whether we try simply 
to evaluate a particular project or subprogram within the overall State rehabilita- 
tion program, we need some guidelines or framework for thinking about programs 
in the context of evaluation. In the following sections we will try to present such 
a framework. 

There is nothing simple or obvious about the selection of criteria. The evaluator 
must be aware of what is to be measured, what measurements are or might be 
available, and how strong the relationship is between the two. In the realm of 
social programs, especially, the effects to be measured may be difficult to quantify. 
No single criterion covers everything. The use of single criteria might distort eval- 
uation. Thus, there is a need in rehabilitation for multiple criteria . 

A m^jor barrier in evaluation of rehabilitation activities is the lack of clear-cut 
criteria or excessive reliance on single criteria. The right questions must be asked 
in order to reach the right answers. Generally, there are different levels of objec- 
tives: Short-term, intermediate and long-term. These levels are related to the 
chain of assumptions which are thought to result in a long-term effect. For ex- 
ample, the objective of providing training to disabled persons is based on assump- 
tions that training will lead to increased employability and to eventual improve- 
ment in income and employment/ Specific output measures must be defined 
for^ach level of objective. 

In vocational rehabilitation, for example, is ^ client "rehabilitated" when he is: 

(a) Fully employed 

(b) Fully employed in a job he was trained for 

(c) Fully employed.in a job he was qualified for 

(d) Fully employed in a job hj is satisfied in 

(e) Fully employed in a job and capable of supporting himself and 
his dependents 

(f) Partially employed 

(g) 'Able to care for himself 

(h) Psychologically well-adjusted to his disability 

(i) / Off the welfare rOlls? 

Also, how long must employment last - for six months - for 3-5 years when the 
taxpayers' investment will be repaid - or for the rest of his healthy working 
life? Tfie task may be to measure qualities such as "happiness," or "well-being;' 
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but standard scales for siuh measurement arc lacking. The evaluator must be 
explicit in his assumptions > interpretations, and manipulations of data in formu- 
lating criteria that purport to measure" these qualities. 

T ypes of criteria 

There are several different t>pes of criteria by which the success or failure of a pro- 
gram can be judged. Although' there is no uniform set of criteria by which all pro- 
grams are evaluated, it is useful to categorize criteria for evaluating rehabilitation 
programs. - # 

(1) Client - Community impact 

In addition to question^ concerning client benefit, you can also 
ask questions about the success of a program in terms of very 
broad social goals. What amount of total need has been met? 
Have community*attitudes been affected by the program? 

(2) Program efficiency and effectiveness 

(a) Efficiency assesses the relationship of program inputs 
to outputs. This type of evaluation depends more on 
relative measures than absolutes. Efficiency is judged 
by several kinds of performance criteria related to the 
cost of achieving an outcome and to the sequence of 
events that must occur to achieve the expected out- • 
come. A distinction should be made between lower - 
level and higher-level efficiency criteria. Lower-level 
efficiency criteria deal with questions of use of time 
and resources, only in terms of input, such as case 
flow through time (measured by statistical analysis). 
Higher level efficiency questions concern both input 
and output variables, such as the program's net bene- 
fit to society or taxpayers, given all inputs and out- 
puts (measured by cost-benefit analysis). Efficiency 
questions include: How are resources being used? 
How should the cost of services be shared? Can 
the same results be achieved with lower costs? How 
does the ratio of costs to benefits compare with 
alternatives, or standards? Has time been used 
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efficiently? Have the necessary events taken place 
to achieve the expected outcome? (case flow) 

(b) Effectiveness - the criterion of success is the perform- 
ance of a program as judged by predetermined expec- 
tations of effect. Here program output is measured 
by outcome or benefit criteria which must be based 
on a clear-cut statement of objectives. Effectiveness 
focuses on the output of the program. At issue is the 
performance of the program as judged by holding up 
the results to the expectations or objectives. Effective- 
ness issues are: What Was the effect of program activi- 
ties on outcome? What was the effect of other 
activities on outcome? Why did the program succeed 
or not succeed? 



(3) Program management 

The criterion of success here is the quantity and quality of pro- 
gran} effort. Th&is an assessment of program input and program 
performance. Typical questions are: Is the program proceeding 
as expected? How does the program effort compare with local 
or national standards \rith respect to number of staff, money 
spent, staff assignments, amount of grants obtained, etc.? The 
emphasis is on the form of the program, rather than lit^ftmction- 
ing. *0 
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CHAPTER III 

PROGRAM EVALUATION: WHOOO ... IS TO DO IT? 





There are many organizational models for program evaluation, and the size of 
the agency will dictate the organizational patterns of these units. This unit should 
be clearly defined in the State agency organizational structure. The Director of 
this unit should be at the highest staff level responsible pn a direct line to the 
State Agency Director. The Evaluation Unit should have an independent assign- 
ment to measure and assess the realization of agency goals and should serve as the 
focal point for this activity. x 

The unit charged with program evaluation should be responsible for both program ? ' 
evaluation and program and policy analysis. It will observe current program opera- 
tion, identify prograiii strengths and weaknesses, and present alternative action 
plans which will provide program direction: * 

Generally, the staff in a Program Evaluation Unit should have professional exper- . 
tise in several areas such as: State agency operations, evaluation and experimental 
design, data collection, fiscal and budget activities, statistical analysis, manage- 
ment, and economics. At least one member of the evaluation staff should have 
vocational rehabilitation field experience, preferably at the rehabilitation counse- 
lor level. 
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It would be advantageous if most evaluation staff members had vocational reha- 
bilitation field experiences. 



It is possible to identify certain staff functions which relate to program evaluation! 
Some of these are: 

(1) Director or Coordinator of Program Evaluation. 

(2) Researcher. 

(3) Technicians in evaluation, measurement, data processing, budget and fiscal 
operations, report writing and statistics. 

(4) Support staff. 

It is desirable, and may be necessary in small units that the staff members in an 
evaluaticfi unit have skills in more than one of the above areas. 



Other resources available to evaluation units 

Consumer participation is a vital resource to vocational rehabilitation program 
-evaluation. Consumers should be involved in identifying unmet needs; in provid- 
ing a measure pf consumer satisfaction, in measuring effectiveness of services; 
and can impact program direction. Consumers of vocational rehabilitation ser- 
vices include, among others, referral source representatives, clients and employers. 

Persons with. specific skills or expertise not available in program evaluation units 
" can be utilized as consultants. Consultants can come from outside the agency or 
• from within. Outside consultants usually may be more objective about specific 
evaluation activities. 

A consujtant from within the agency can offer certain specific advantages since 
he is thoroughly familiar with the program and he has clearly established lines 
of communication within the agency. In addition, program evaluation is most 
relevant when it builds upon and involves the people directly affected . 



^ State agencies need to develop the ability to respond to and take advantage of 
data generated by other levels of evaluation. The States must identify thcii ioles 
in program evaluation and be responsive to the issues arising from evaluation 
results. It is suggested that the Federal level be encouraged to assist States by: 
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(1 ) Providing funds to assist States in the costs of setting up a pro- 
gram evaluation unit. ' 

(2) Making available the resources and funds to provide training to 
program evaluation staff in the States. 

(3) Establishing demonstration programs where common evaluation 
designs and models can be developed for \jse by States. 



30 



1 



CHAPTER.IV 

PROGRAM EVALUATION: WHAT IS DONE - DESIGN 

To handle the increasing variety and complexity .of vocational rehabilitation pro- 
gram problems, many evaluative techniques developed by industry and educational 
institutions are worthy of mention. Each evaluative technique has its special use, 
and care must be taken to select the correct technique for a particular evaluation 
activity. The agency director as well as the evaluator has a role to play in selecting 
the technique for evaluating program activities, and the better he understands the 
range of evaluative possibilities, the more likely it is that the vocational rehabilita- 
tion program evaluation will bear fruit. 

The selection of a method of evaluation depends on many factors, the activity to 
be evaluated, pie relevance and availability of historical data, the degree of poten- 
tial accuracy , the time period to be evaluated, the cost/benefit (or value) of the 
evaluation to thVagency and program, and the time available for making the 
.evaluation cycle A 

These factors must be weighed constantly, and on a variety of levels. In general, 
the evaluator should choose a technique that makes the best use of available 
resources and data. If he can readily apply one evaluative technique of acceptable 
accuracy , he should not try to "over-kilT by using a more complicated and advanc- 
ed technique that offers potentially greater accuracy, but that requires nonexist- 
ent or difficult to acquire information or information that is costly to obtain. 

The main purpose of this chapfTriTto present an overview of the way a vocational 
rehabilitation agency may approach an evaluation activity, and explain how to 
match an evaluation method to a program activity. 

The subsequent charts present several examples of techniques used in program 
evaluation^ The charts are by no means complete and exhaustive, but they do 
provide a body of basic information about the different kinds of evaluative tech- 
niques. Some of the techniques charted are not in reality a single or complete 
method of evaluating program or agency activity. They are to be interpreted as 
descriptive of the basic concept of each technique. 
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It is also appropriate to discuss the evaluative activities designated and some activi- 
ties beyond that outlined in the charts. The need for program evaluation in State 
Vocational Rehabilitation Agencies is not for better evaluation methods, but fdte> 
better application of thfevaluative techniques at hand. 



The. Delphi technique is a procedure for systematically eliciting and refining the 
judgment of a group of experts. Generally, this technique involves: 

(1) Obtaining opinions from experts by use of a questionnaire. 

(2) Controlled sharing (or feedback), and reformulation, of the results 
among the participants in the group. 

(3) Aggregating individual opinion into an overall group judgment. 

A modification of this approach could be used to elicit opinions from participants \ 
at yshious levels, including the client level in the system. 



f Program rescagft - Evaluation of overall program efficiency 




There are several general techniques for assem- 
bling the information to conduct such an overall 
program evaluation of efficiency . Each tech- 
nique really assesses Jthe program in terms of 
very different criteria. The criteria of three ex- 



ample techniques are: 



( 1 ) The length of time before" the client "pays back" the public for their 
tax expenditures on his rehabilitation. Such repayment takes the 
form primarily of ta^s paid by the client as a result of his increase 
ed income and of savings in institutional costs and welfare payments 
which the government would, in-the absence of rehabilitation, have . 
incurred. 

(2) The net increase in real income of clients and client satisfaction 
from government programs and employment. 

(3) The most favorable ratio of social benefits to social costs among 
alternative programs and strategies. This approach is more prop- 
erly formulated - in the economist's language - as maximizing 
net present value of social benefits. 



<tfte ^rst approach looks at the program from the perspective of the taxpayer. The 
secbnd approach views the program solely from the perspective of the client. The 
last approach, the one most generally known as benefit-cost analysis, tries to 
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aggregate benefits and costs over society as a whole. Often the latter approach is mis- 
used and the evaluator focuses only on those benefits and costs which can directly be 
measured in monetary terms. * 

The program perspective of the taxpayer or payback period model analyzes returns 
and costs borne by taxpayers. This model may be particularly useful in rehabilita- 
tion since the major cost of services is financed by non-client taxpayers, while most 
benefits of increased earnings are enjoyed primarily b'y the client recipient of ser- 
vices. Because the payback period approach concentrates on the net return to tax- 
payers, it can be an effective tool for showing legislatures and government execu- 
tives the value of investing more resources in rehabilitation programs. Few, if any 
other social service programs represent such a good investment for the taxpayer, 
and this is true even for programs serving the most severely disabled and hard-core 
public assistance cases. Indeed, payback period analyses often show that the tax- 
payer's return is greatest in investing in these more difficult cases, since the taxpayer 
might otherwise be supporting these individuals for life on the public welfare rolls. 

The technique of "looking at the efficiency of the program in terms of the client's 
experience is probably the least applied of the three approaches to measuring 
higher-order efficiency and overall value of the program. The technique views as 
benefits the increase in client earnings- However, reductions in welfa^ payments 
as a result of increased earnhgs and income are viewed as a negative benefit. Simi- 
larly, program costs are not considered at all. Rather, the perspective of the client 
is concerned solely with foregone earnings while in the rehabilitation process and 
the costs borne directly by the client and his family. 

The value placed by the client on reducing his state of dependency becomes very 
important. The value of considering thi^erspective is the insight it can give in 
understanding why and how clients may respond to various kinds of rehabilitation 
services. -v. 

The benefit-cost model is the most commonly applied. The technique is subject 
. to many pitfalls in practice*. The evaluator may choose to look and emphasize 
only those benefits or costs which are easily measured and valued in monetary 
returns. Readers of the analysis may focus only on benefits and costs based on 
"hard data." Agencies can be motivated by such analysis to focus on providing 
services only to those clientswho provide the "greatest return," rather thanking 
such analysis to evaluate alternative strategies and programs for rehabilitating par- 
ticular disability groups.' The results'of the analysis can be highly sensitive 4b par- 
ticular asWptions which are made, and these assumptions and their sensitivv^ 
^reSjten not made explicit. As the analysis extends to valuing benefits (e.g., 
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homemaking) which are not directly measured through market-set prices, agencies 
could conceivably adjust assumptions on valuing benefits to justify virtually any 
program, however inefficient or ineffective. The real need in cost-benefit analysis 
is to establish conceptual models with commonly accepted assumptions which most 
State agencies would employ, so that the results of the analysis would be compar- 
able and changes in assumptions could readily be identified and assessed by readers 
of the analysis. v • ■ 

* 

Program research and inquiry 

Evaluation of work stability is best done through follow-up studies of clients some 
time after case closure. This is best done with a sample design drawing clients' 
names randomly from R-300 records across the full fiscal year. Clients are best ' 
contacted by phone or in person. Mailed questionnaires can be informative, but 
the biases in response are usually quite significant, since the overall response rate 
may be less than 50% or even 25%. The characteristics of clients who do not return 
mailed questionnaires must be carefuHy analyzed, and the generalization and inter- 
pretation of questionnaire information modified to reflect such response biases. 



Program research - Client impact evaluation 

Methods of obtaining such information on client impact include: 



( 1 ) Adding questions to State agency- reports to be completed by 
samples of clients or all clients at case closure. 

(2) Follow-up studies of clients sometime after case closure. 

(3) Group sessions with clients and/or their families. 

(4) Assessments of State advisory groups comprised of former 
clients and representatives of client organizations. 

{5) Survey of client satisfaction at each closure exit (08, 26, 28, 30) 
should be done at frequent intervals. 

(6) Employment stability survey of clients rehabilitated several 
years after closure. 

(7) Survey of employer satisfaction. 

Overall client impact can also be measured by experimental and program research 
techniques. Such techniques deal with how much of the measured change is actually 
attributable to the client's recelpTof rehabilitation services. Sometimes this con- . 
ccrn is also rephrased as the question: How does the client's current experience . 
(after closure) compare to what would have been his experience, had he or she never 
received rehabilitation services? 
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Qutside consultants and judgment by experts 

The outside consultant may be able to help evaluate that which is obvious to an 
outsider but not so obvious to people within the organization. Tradition does not 
justify inefficiency and mediocrity. The agency that is set in its ways and has deeply 
ingrained problems is the organization that needs the consultants the most. 

In evaluations done by a consultant the all important point is not whether you 
agree with the findings but whether or not they are well documented. If the con- 
sultant is carefully chosen, you will see your agency through the eyes fff an objec- 
tive outsider. 

Judgment by experts, although the least objective, is one of the oldest techniques 
of evaluation. Expert opinion is useful in selecting among several alternative 
courses of action. They are also useful when there is a Jack of objective or theo- 
retical knowledge that would clearly single out a preferred course of action. Ex- 
perts ujay be from within or outside the rehabilitation system and may be either 
specialists or generalists. Expert judgment may be based on the application of 
Existing theories or on intuition. There may be factual judgments and value 
judgments. 



Observation - Visionary evaluation 




The process of doing an evaluation by observation 
can prove to be tedious but not necessarily diffi- / 
cult. There are two important factors to be imple- 
mented in using thirmethod of evaluation. They 
are: 

(1) Have the observer include sufficient 
detail in his records. 

(2) Have the observer prepare permanent 
records imtriediately after a day's 
observation is made. 



lie principal advantage of direct observation is that it culminates in,a highly detail- 
ed and nearly complete record cff a person's actual performance. It does not depend 
upon his ability to interpret a questionnaire correctly or upon his memory of a not 
very important and perhaps not very recent event. It is-not influenced by any tend- 
ency to rationalize his behavior or to make it appear in the best light. 
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A second, but at times equally important, advantage is that it occasionally pro- 
duces ideas that can be tested at a later date. 

A disadvantage of direct observation in evaluation is that it provides informa- 
tion on behavior only. Behavior cannot always be easily interpreted. 

Another disadvantage of the observation method is that the results can be biased 
by when and where the observations are made and the personal prejudices of 
the observer. 

Two additional problems that this method of evaluation creates are: 

(1 ) The observer has no control over important variables. Cause 
and effect are sometimes indistinguishable. 

(2) The reports are narrative rather than quantitative. 

All the disadvantages noted are important, but they are not necessarily negative. 
The important\question is: "Can the limitations of the method be accepted con- 
sidering the information it yields?" 

Evaluation through historical and statistical analogy (manageme nt measure- 
ment procedures ) 

Statistical analysis and comparison of program inputs against professional or gov- 
ernmental standards are typical means of measuring management criteria. 

Statistical analysis is the most common and often the sole technique used for 
measuring.program criteria. Useful statistical measures include the mere order- 
ing of obSrvations (ranking - better or worse, more or less), the use of weighted 
averages (mean, median, mode), the distribution of cases (standard deviation, 
variation), and making comparisons (correlation, factor analysis, analysis of vari- 
ance, nonparametric probability statistics measuring strength of association chi 
square and statistics measuring nonrandomness). Statistics can show the quantity 
of effort expended, imbalances in services to certain groups, the movemeijt of 
clients through the rehabilitation process and so forth. Statistics provide gross 
* data useful in pinpointing problem areas or areas in need of furthej study. To 
fully .understand problems and their causes, higher-order measures! or indepth 
investigation into particular cases is necessary. . 
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Standards 

The Rehabilitation Services Manual, Commission on Accreditation of Rehabilita- 
tion Facilities or standards of practice outlined by professorial groups are ex- 
amples of models for evaluation of program activities. Data a?es^>llected and 
analyzed on factors suggested by the model and conclusions drawn. The limita- 
tions of this approach are based mostly on possible inadequacies of the standards 
themselves. Standards may lack comprehensiveness. They may be dated in terms 
of their representation of current reality or be based on generalities not applicable 
to all individual cases. For example, to.measure success in terms of the number 
of rehabilitations per 100,000 population may not accurately represent large 
States, Also, this figure assumes that the incidence of disability is the same across 
all States. This may be a false assumption. 

Another requirement for good administrative support of rehabilitation is a work- 
ing information system. An information system will make pertinent data'avail- 
able to management in tfte shortest period of time and is necessary to support 
any administrative structure. Not only must adequate records be kept on case 
histories and treatment given, but also details of referrals, costs of delivery, and 
follow-up efforts should be well documented. Subjective reports on the progress 
of individual clients should supplement ratings of vocational achievement. Even 
more important, a management information system should link costs to client 
records, performance measures, and services received. This informs the evaluator 
of performance per unit cost and provides information which is of ready use in 
evaluation of management. 

Field experiments or demonstrations 

Field experiments and experimental demonstrations are two research designs for 
testing the relationships in the natural setting of ongomgfcrograms. These are 
widely used in social science research because of the difficulty on controlling 
factors in the human sphere. In the field experiment there is control of some 
factors without interfering with the normal daily routine of clients. 

The evaluator either controls the persons who are and who are not to be exposed 
to the program by changing an aspect of a program; e.g., by controlling workload 
of two different staff groups, or by changing worker assignments. The demon- 
stration differs from the field experiment in that the social setting is changed 
by the program administrator rather than by the evaluator. In demonstration 
efforts research goals are generally of secondary importance. 
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In utilizing these designs errors may result from: inappropriate topic for inquiry, 
conception of a faulty experimental design, or failure to introduce or to retain 
appropriate controls. Successful field experimentation and demonstration rely 
on careful advance planning. 

Evaluation thrpugh an experiment model 



. there are generally considered to be nine categories of experimental and quasi- 
experimental designs. These range from the one-shot case study or "after-only" 
study, one group pre-test, post-test (the recipient(s) is measured before and after 
' administration of treatment)," to the pre-test, post-t est, control group design. 
(There are two randomly selected, equivalent control and experimental groups. 
A "before*' and' "after" measure is made of both and comparisons made.) The ' 
latter design is the classic true experimental design and isjthe*"strongest" in 
terms of the degree to which variables are controlled and unbiased. The one-shot 
case study , altjttjugh the most commonly applied design in evaluations of reha- 
bilitation programs, is the weake.sk 

In rehabilitation it is rarely possible to obtain equivalent control groups, since 
this would be difficult to arrange. Another alternative is to compare rehabilitation 
with other programs serving the disabled, eliminating a 'control group which re- 
ceive no services. Indeed, it has also been argued that it is misleading to believe 
that control groups receive no treatment. The most commonly used designs are 
the "one-shot" follow-up study and "before and after" design. This approach is 
weak as far as pinpointing specific factors that contribute to rehabilitation. 



The expcrimentalHiodel ideally involves five procedures: 



(2) 

(3) 



Definition of the target population. 
Drawing a representative sample. 
Allocation of the sample at random into experimental 
afld control groups. 

Administration of the program to one group and not to 
the control group. 

Comparison of resulting differences between the two 
groups. 





Survey 

By definition, survgy. means to look over - to see to examine as to condition, 
situation, or value - to appraise - to inspect - to scrutinize. 

The survey is perhaps the most commonly used method of collecting information, 
other than statistical data, for the purpose of establishing facts, trends, or opin* 
ions It is most frequently used in the.form of a written questionnaire, although 
at times it kdo'ne verbajly. The method utilized will depend upon the.informa- 
tion being sough^b* particular preference of the person doing the survey, and 
often upoittl/amount oftoa* available to get the desired results The advan- 
tage of a written survey is that the'infognation can be reviewed at the conven- 
ience of the surveyor. T" 

The survey iwftasic tool of the evaluator. An important thing to keep in mind 
is that the method of questioning strongly influences the responses elicited- 
Therefore, the effective evaluator is one who learns how to structure his result- 
getting survey. 

It is important that the survey reflect a sense of partnership. Employees and 
clients like to know that their, responses not only present facts but also gives 
them an outlet for the expression of their opinions. . : 

In developing a survey questionnaire, it is vitally important that the evaluatoi? j 
know precisely what he is evaluating. His questionnaire should elicit a complefe 
willingness to respond accurately and thoroughly. Information that is already/ 
known or obtainable thrCifch other means should not be a part of the survey/ 
The evaluator should kn\w^hat information he needs and what the best so&rces 
of this information are^ 

Inputoutput model and ev aluating case flow 

The rehabilitation process begh* with referral and proceeds through evaluation 
and diagnosis, eligibility determination, development of vocational objective, 
development of plan of services, provision of services and ends with some type 
of closure, preferably because of satisfactory employment. This is th^standard^ 
against which program indicators are judged', ^he task is to determine howptUal 
practice deviates from the standard. \ \ J . r ~~ 

The movement of Individuals through the^chabilitatidnprocess is defined as 

case flow. . 

l 
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Case flow information can give an indication of whether the program is proceed- 
ing as expected. 

The experienced vocational rehabilitation counselor annually closes clients at 
ahout the same rate as new clients enter his client load. The total number of 
clients closed by the experienced counselor may vary from year to year depend- 
ing upon the size of his client load. But the percentage of cases closed from each 
closure exit of the total closed from all exits remain relatively constant. 

Another computation which broadly measures client flow is to determine each 
•year the ratio of clients closed in all statuses to the total clients involved in the 
program. A ratio of .500 or above reflects the agency is either in balance with 
new referrals or there were more cases closed than entered the case load Ratios 
have meaning when adjustments are made for case composition and when com- 
parisons are then made with past performance, the national average, or other 
states. 

A program evaluation unit may observe and study trie total State caseload, the 
client's progress through the, statuses, and his exit from the rehabilitation process. 
Evaluation focuses on the flowyof the client population through the process and 
the choice and speed of services delivered to the client . The time a case is in proe- 
ess from referral to clqsure and the balance of clients entering the process of 
those exiting are benchmarks which give a general overview of the effectiveness 
of program operations. 

Yet. & is important to remeinber that whether the client received what he needed 
as qukfciy' is possible, may not measure quality of service. Thus, there is a need 
for combination measures and for the periodical review of a sample of cases. 

•/ 

'1uso critical to program evaluation is an examination of the individual's progress 
through the various stages (statuses) from referral to closure. The length of a 
* client's stay in any one status partially reflects the counselor's ability to guide 
the client through the many.services needed to affect his rehabilitation. The 
assumption, which is borne out by cost data, is that an unusually long period of 
time in various statuses often indicates that resources are no; being effectively 
used. The client's goals-are not being achieved, and the probability increases • 
that the client may drop out in frustration. The counselor is often expending 
considerable amounts of his own time and energy and of case service money in 
efforts which are not producing results. 
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le State agency, to perform evaluation, must develop the abilitv and expertise 
! to measure client flow at frequent intervals The first step in developing this 
Ability is to identify those points in the rehabilitation process that require major 
decisions by the counselor and client 

\ 

fW example, a quarterly analysis might be done of the How of cases through 
strategic points of the rehabilitation process, such as referral and applicant status, 
00\ 02. Such an analysis might take the form of* * 

( 1 ) Analyzing clients in status 00-02 (referral: applicant) three , 
months or longer. Research shows a negative correlation be- 
tween the length of titne which a case stays in status 00-02 and 
a successful closure status 26. 

Analyzing clients in status 10 (Plan Development) and 12 (Plan 
Completed). The number of months a client remains in status 

^mnsdgs^s^biiity to niake decisions 
; crucial flninltfin the j**fSMitatftH^ocess. 

Analyzing clients in status 20 (ready for employment) and 
24 (service interrupted). Tht* length of time a client is in 
status 20 mpy refleta on the quality and choice of services 
planned and bnplem^ed- If the client remains in status 
longer than three raor^s. the services rendered may not 
» been adequate. Direct counselor, intervention may be 
nejessary for placement purposes. If a client stays in status 
2f three month? or longer in most instances, the client 
fould be rephased through the rehabilitation process or 
^closed through one of the closui^ exits. 




Leading indicator 

Indicators exist that can be used to asse^several aspects of management. For 
example, a large percentage of staftnr30 closures (closed before rehabilitation 
plan initiated) or status 28 closures (closed, unemployed after plan initiated) 
may indicate things like: High counselor turnover, an inactive caseload, lack of 
client service funds, work performed by an in^perienced counselor or support 
personnel, insufficient data upon which to detehpine eligibility, and improper 
status classification which more appropriately shofydLhave been directed to 
status 04 or 06, extended evaluation. An annual evaluation of $t3tus 28 and 30 
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closures offers extensive information concerning the rejection of clients in the 
State agency. Did the client drop out because he found a job on his own, be- 
cause he was dissatisfied with Jus plan, because he feared loss of his welfare sup- 
.port, or what? 

Other causes and indicators of program imbalance are: 

J 

(1 ) Input ^ceeds oltput - number of new cases is greater than 
the-fromber clostd fronrall categories of the vocational reha- 
bilitation process. / 

(2) Increase in new ( referrals over the counseling staffs ability 
to process theni. 

(3) Lack of experience of the counseling and support personnel. 

(4) A critical budget imbalance for various program services. 

(5) Lack of funds. 

(6) High staff turnover rate. 

(7) Over-extended program expansion - expanding programs at 

a faster rate than the capacity of the agency to deliver sendees. 

- (8) Radical change in program direction or priority. 

(9) Management and organization constraints - regulations,' inade- 
quate supervision and/or administrative direction. 

All or any one of the above factors can contribute to the inability of an agency 
to function optimally. Many of the factors outlined above are external and 
may result from any number of circumstances, such as legislative mandates, 
rapid population increase, inadequate tax support, and increased awareness of 
community health and social problems. 
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ife-cycle analysis 



The life-cycle analysis can be used when a hew program is initiated. In this 
approach the evaluator tries to evaluate this new program with a similar, older 



\ 

program, whose overall pattern shoflld be similar to that of the new program. The 
assumption here is that both programs should have a similar program pattern. 

Federal reporting and performance statistics - A tool for evaluation 

MThe Federal programs provide case service re- 
ports (R-300) and many other statistical ab- 
stracts. The R-300 form not only provides 
basic information to the rehabilitation prac- 
titioner, but it is also a basic tool of the e valu- 
ator. The questions on the R-300 lend them- 
selves to brief, precise answers. 
There is, of course, room for error since the 
R-300 is completed by humans, and they are 
fallible. Even the most careful and precise 
individual has a chance of making an error. 
Nevertheless, it remains a most factual document. 
The R-300 is designed to take us completely 
through the rehabilitation process. Thus, it provides the evaluator with a rather 
complete source of data. It readily lends itself to evaluation at many stages in 
the rehabilitation process. 

The data that can be retrieved from the R-300 often is the most readily accessible 
data available in an agency. It not only provides the agency with the information 
that is necessary for reporting to the Federal Government, but it can provide the 
State agency with an overall picture of who it is serving, at what costs, with what 
results, plus time expended, and the data is identical with all States reporting to 
the Rehabilitation Services Administration, thus facilitating State Agency com- 
parisons. 7 

The design of the R-300 is such that it lends itself to review by either humans 
or computers. The information contained therein permits ongoing evaluation 
without the necessity for additional forms or people. 

There are limitations, however. It does not measure such things as case record- 
ing, subjective observations, attitudes, or feasibility of services, just to mention 
a few. The evaluator seeking this type of information must utilize other methods 
of evaluation .v. 
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In using the R-300 when doing an evaluation the evaluator can limit himself to 
one factor or he can evaluate many factors. He can evaluate one counselor or 
the entire counseling staff ...He can evaluate, one day's work or cne year's work. 
The range of things that can be evaluated through the R-300 is limited only to 
the imagination of the evaluator. 

Some typical Federal publications using data from the R-300 case service report 
are: 

(1) State Vocational Rehabilitation Agency Program Data (pub- 
lished annually by HEW) 

(2) Caseload Statistics, State Vocational Rehabilitation Agencies 
(published annually by HEW) 

(3) Statistical Notes (published monthly by HEW) 
Client feedback 

The need for client feedback into the rehabilitation pregVam should be maxi- 
mized. The client perspective is highly valuable as a resource for program im- 
provement. It would be desirable if State agencies would routinely question 
or sample clients to determine the client's opinion of tiie services he received. 

Such questions might include probes on: 

(1 ) Whether the job in which the client is employed makes use 
of the training he received as part of his rehabilitation plan. 

(2) Whether his employment or other status at a 26 closure re- 
flects that the needs for which he came to the Division or 
of Vocational Rehabilitation have b^en met, 

(3) What other problems does he foresee that might interfere % 
with his keeping his job. 

(4) His or her assessment of improved personal capabilities in 
non-job activities as ? result of the rehabilitation services 
received. 
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(5) Changes in the employment status of other family members 
during the rehabilitation process as a result of services re- 
ceived. 

(6) His evaluation of the quality and sufficiency of services 
received, and of any difficulties or problems encountered 
during the rehabilitation process. 

(7) The amount of money which the client and his family may 
have personally paid for services, etc., during the rehabili- 
tation process. 

(8) Any savings in medical, child care, housekeeping, attendant, 
or other costs which the client and his family have achieved 
as a result of the client's improved Capabilities ^ 

(9) Services received by the client from agencies other than those 
recorded in his rehabilitation plan 

(10) CUfent suggestions for improving services to future clients. 

(11) Client willingness to participate in consumer organizations 
working with rehabilitation agencies and future clients. 

* 

Individual case review as an evaluative a ctivity 

f 

Maintaining the integrity of the records is the responsibility of the counselor and 
the first-line supervisor. „ * 

Individual day-to-day case review is not a function of program evaluation but a 
responsibility of personnel who supervise casework review and management. By 
case review we mean the activity of opening individual client case folders and 
reviewing them to determine conformity with State and Federal governmental 
standards and good professional practice. 
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CHAPTER V 



PROGRAM EVALUATION: AN INVITATION TO INNOVATION 



^OVATION THRU 
. NOWAM EVALUATION 

WHEN-- NOW 

IwewyouragencyI 



The purpose of this chapter is twofold. 
The first part will deal with suggestions for 
implementing the findings of a Program 
Evaluation Unit, and the sefcond part will 
be to examine some considerations of im- 
portance in attempting implementation. 



We all know how hard it is to change our 
own ways and to get people to change 
theirs. When one goes about trying to im- 
plement change of any kind, the person 
responsible For effecting the change should 
be fully aware of these resistances to change. 
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Program evaluation utilization 

An agency wishing to take full advantage of program evaluation must have built 
into its system a structured ongoing procedure that helps to insure that the re- 
sults of valid evaluation are given serious consideration. It is evident that very 
often the results of quite valid evaluations are lost or not acted upon. Perhaps 
the reason for this is because the agency has not set up functional utilization pro- 
cedures. It is recommended that the procedure be formal and structured. 

The results of an evaluation should be presented in a written report that clearly 
states its conclusions, and data upon which the conclusions were based. Recom- 
mendations should be included where indicated. 

The initial report would be sent to the agency administrator or his designee. 
Copies of the report would then leave the evaluator's desk earmarked for review 
by agency personnel directly responsible for the program under evaluation. A 
response should be expected from the program personnel within a reasonable or 
designated time period. 



The program personnel should respond to the report and would have an obliga- 
tion to mflke recommendations relative to the need for changes and additions 
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within the program. This response should be sent to the administrator's office 
for policy and administrative consideration 



Perhaps the most important part of this procedure is the responsibility now facing 
the office of the agency administrator. " [ 




It is the responsibility of the program evaluation unit to make the state agency 
director fully aware of the program evaluation findings/ The director's response 
to ttie findings of an evaluation.report could be the single most important factor 
in its Implementation. 

Once a program administrator has made some decision in response to the find- 
ings of program equation, there is still a need to nonitor the implementation 
of that decision. Too often, decisions are made, but the follow-through on im- 
plementation is inadequate. 

The responsibility for monitoring should be explicitly assigned by the agency 
director, along with the requirement for a report back to his office within 
time period . 

Without the support of the agency director, program evaluation can never become 
an ongoing, effective source of agency innovation and improvement. 

Evaluation 
Agency and Administration Cycle 



(Monitoring) * 

IMPLEMENTATION ^^^^ \ EVALUATE 

REMEDY 




(COMMENTS) T * (CRITIQUE) 



RESPONSE \ / REPORT 
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Dynamies of organizations and progrgfn evaluation implementation 
The outline for this section willinclude: 

(1) Organizational constraints to change, and 

(2) Other factors that should be kept in mind when trying to 
implement a program evaluation unit into vocational reha- 
bilitation operations. 

Organizations and change 

It is possible to identify some organizational characteristics which constitute con- 
siderations for anyone trying to effeci change in an organization. 

There are at least four types of characteristics 
which all organizations are likely to possess. These 
characteristics are. Survival, s^bjhty, Eurgose, and 
membership . 

Survival is probably the first commandment for organizations as well as for people 
Some activities or information from outside the organization may cause problems 
in the form of exploitation, subversion, take-over, or contamination. All organi- 
zations need to maintain the ability to intercept these messages. When organiza- 
tions become especially concerned about survival, this interception may keep out 
virtuallvsall information from the environment. 

Stability is an expressed need of all organizations. To maintain themselves and 
carry out their functiohs^they must maintain some sort of internal steady state of 
equilibrium* Information which is new in .content may be upsetting, constituting 
a threat to the established equilibrium. 

Nearly every organization is bound together by some common purpose - usually 
< a purpose which extends beyond survival and stability. To hold to this purpose 
they must define and establish the parameters of vocational rehabilitation pro- 
grams, and agencies will avoid inputs contrary to defined practices. 

Fourth, it should be noted that the fact of membership in an organization creates 
a vsry complex $nd many-sided barrier to outside influences. Those who share, 
common membership in one organization are also likely to share many attributes 
such as a common language, a common set of values, a common ideology, and a 



-52- 



common perception of role and status- All these attributes which tie them to- 
gether also set them apart from outside organizations. 

1 

There are many factors related to the necessity for limiting inputs into an organi- 
zation. Some of the factors reducingrtaowledge flow into an organization appear 
below: r 

(1) The need for stability 

Many writers have addressed themselves to the general impact 
of order and constancy of knowledge flow. Most organiza- 
tions by their very nature ase conservative, and there, is a con- 
stant concern in organizations to maintain internal stability. 

(2) Use of coding schemes 
Members of an organization which requires loyalty and com- 
mitment tend to acquire common coding schemes or shared 
ways of articulating the things relevant to them. Organiza- 
tional coding schemes can be a determinant of communica- 
tion, in that they distort, reject, accept, anxl transform what 
is said. A group establishes its own particular identity by 
enlarging its uniqueness. One way of establishing this unique- 
ness is to define^ocabulary peculiar to the group. A 
unique coding scheme or vocabulary thus constructed tends 
to make outside communication with the organization dif- 
ficult. 

(3) Fear of outsiders 
Knowledge from the outside of an organization can many 
times be viewed as a threat to it, not only in terms of up- 
setting the orderlinesses a consequence of a deliberate 
change ,,but also as a direct maligning of the organization 
and its members. 

(4) Personal threat 
Related to the suspicion of outsiders as a threat to the 
organization as a whole is the belief that outsiders will say 
or do something that will harm an organizational member. 
Newman (1958) has noted that behavioral scientists are* 
many times refused admittance to organizations by mem- 
bers who think whatever the scientists generate willbe an 
indication of member failure. 
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(5) Economic conditions 

The economic situation of an organization has a great deal 
to do with the knowledge it accepts and utilizes. If an 
organization has a very solid growth pattern and financial 
situation, it can afford to seek out new and uncertain dis- 
coveries and innovations for experimentation. 

(6) Size 

Research on the impact of organization size on information 
flow is quite consistent. Mansfield (1963) found that larger 
organizations adopt new ideas and technology at a faster 
rate than smaller organizations. Most of the research done 
in this area coacurs with this point of view. 

Now that we have enumerated some of the barriers to the input of new krtowledge 
into organizations, it is appropriate to turn to some mechanisms that an organiza- 
tion uses to overcome such barriers. A person responsible for trying to implement 
a program evaluation unit into an agency should keep the following considera- 
tions in mind: 

(1) Perceived reward value 

Perhaps the most fundamental motive for seeking new knowl- 
edge is its potential reward value. An organization's speed of 
response to a change is directly proportional to the amount of 
value perceived by the organization. 

(2) Perception of crisis 

The perception of great difficulty in an organization usually 
results in a hurried search for help. Thus, a crisis can stimu- 
late knowledge flow inside and outside an organization. 

(3) Examining other organizations 

A person interested in effecting change in an organization 
can facilitate knowledge entry by sending a member out- 
side to procure new knowledge from other relevant organi- 
zations. There are several forms such outside assignments 
can take, all of which may be considered as types of train- 
ing. Formal academic conferences, courses, conferences, 
seminars, professional meetings and conventions often con- 
tain knowledge inputs that are of great utility to the home 
organization. 
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(4) Training 

Organizational training is a potent means tor encouraging or 
discouraging knowledge flow The training of top and middle 
supervisors and counselors is one of the key factors determin- 
ing the rate of introduction of innovations. 

(5) Importing human competence 

Another way for an organization to increase the infusion of 
knowledge would be to hire a person who possessed the ex- 
pertise and competence it needs, This could be described as 
the importation of human competence. 

The foregoing have been considerations in an organizational sense regarding the 
implementation of a program evaluation\unit into a vocational rehabilitation 
agency's ongoing operations. 

Havelock (1969) has proposed a most interesting 
perception regarding organizational change. It is as 
follows: "Organizations, like people, can be said- to 
have values, purposes, status, size, capacity, etc. 
Many of these characteristics operate to facilitate 
and inhibit knowledge flow in organizations in 

much the same way as they do in individuals 

.... The organizations are composed of people. 
Henqe, in many respects interorganizational pro- 
cesses can be reduced to interpersonal processes," 
(Havelock, 1969) 

With the above quotation in mind, it is appropriate at this time to discuss factors 
affecting ho^eople go about changing and considerations for anyone trying to 
effect a change. 

Individual resistances to change and ways to overcome them 

Personal resistance to change has been studied extensivfely. Goodwin Watson 
(1962) described the stages of resistance to a typical innovation as. 

(1 ) Massive, undifferentiated, few take the change seriously. 

(2) Pro and con sfdes identifiable, resistance can be defined and ^ ^ 
its power appraised. 

(3) Direct conflict; resistance mobilized. This is a crucial stage 
for survival. 
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(4) The changers arc in power. Wisdom is needed to keep oppo- 
sition from mobilizing. Resistors are seen as cranks or 
nuisances. 

(5) Old adversaries are as few and alienated as advocates were 
in the first stage. Advocates now resist any new change. 

Resistance to change is not a single entity or process. But it has many parts 
major features are: 

(1) The change is a threat to the established social structure. 
Innovations sometimes pose a threat to established social 
structure. A general finding of these studies is that resist- 
ance to innovation is roughly proportional to the amount 
of change required in the social structure, and the strength 
of the social values which are challenged. 

(2) An inrtovation can be a threat to vested interests. Generally 
when a ruling minority has vested interests in keeping things 
the' way they are, only token innovation takes place. On 
the other hand, change may be accepted at upper levels of a 
hierarchical organization only to encounter vested interests 
at lower levels in the organization. 

(3) the innovation can be a threat to the individual . Individual 
resistance to change is usually because the person is unfamil- 
iar with the way things will be done when the change is insti- 
tuted. They are content and satisfied with the way things 
are and, hence, resist any innovations that may change their 
day-to-day activities. 

A change also can be construed as being a threat.to the indi- 
vidual's status. When a change appSh to diminish the in- 
fluence or power of a certain group, the change will be 
vigorously resisted. 

(4) Resistance because of the characteristics of the innovation 
Some innovations are resisted primarily because they require 
group acceptance rather than individual acceptance, and the 
characteristics of the innovation-make group concensus diffi- 
cult to achieve. An example of an innovation which has 



encountered widespread resistance because of this factor is 
the universal adoption of the metric s>stcn 

In addition to our knowledge of why people resist innovation, we must also know 
about how people go about adopting and using new information. The following ' 
section will deal with generalizations and principles extracted from the literature 
of how people go about utilizing new information. 

Utilizing new information 

Research has been done in diverse fields regarding the diffusion, dissemination, 
and adoption of innovations. 

This research has yielded some generalizations that one should be aware of when 
attempting to effect change in vocational rehabilitation agencies. 

The first generalization of importance is a finding that people often do not adopt 
new ideas, practices, and products upon first hearing about them. 

Instead, people proceed through a series of discrete, 
identifiable stages in adopting new ideas. These 
stages are 

(1 ) Awareness - The first knowledge about a new 
idea, product, or practice 

(2) Interest - The act of seeking of more exten- 
sive and detailed information about the idea 
to determine its possible usefulness and appli- 
cability. 

(3) Evaluation - The weighing and sifting of the 
acquired information and evidence in the 

• light of the. existing conditions into which the 
practice would have to fit. This stage is some- 
times called "mental trial." 

(4) Trial - The tentative trying out of the prac- - / 
tice or idea accompanied by acquisition of 
information on how to*do it, and 

(5) Adoption - The fulHcale integration of the 
practice into the ongoing operation: 
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A second generalisation of importance for effecting .change is that information 
sources vary in their effectiveness at different stages in the above-mentioned 
awareness-adoption continuum. Impersonal information sources such as research 
reports, articles, etc., can create awareness, interest, and even some mental trial. 
However, in the latter stages of this continuum, personal sources of information 
are often necessary in order to assure trial and adoption. The implications for 
the "change agent" are unmistakable. 

A third generalization that should constantly be kept in mind by the interested 
change agent is that organizations have within them people who are called 
"opinion leaders" (Wilkening, 1952). An "opinion leader" is a p'erson within 
a group, who because of rank, status, role, or personality, is the one looked up 
to by other members of the group. For a new idea to be accepted by a group, 
it often must be first legitimatized by these "influentials." It is crucial that 
these "opinion leaders" be first identified and then utilized in fostering adoption 
of new ideas. ' 

A further generalization that has been proven by extensive research is that the 
target audience or the people who will take on the new innovation must first per- 
ceive the need for the innovation in order to insure its successful integration 
into ongoing practice. In short, the target audience must be involved in the early 
stages of the planning for innovation. If initially involved with the planning for 
innovation, when change does occur, its chances of surviving are greatly enhanced. 

A further generalization is that adoption is easier if the new idea, practice, or pro- 
duct has a clear-cut advantage over that which it is attempting to kplace. Demon- 
stration of a clear-cut advantage enhances the chance of innoviitto^ becoming 
integrated into ongoing practice 

Consistent with the former generalization is the one that the new idea, practice, 
or product should be easily demonstrated. If people can readily see the new way 
of doing tilings and find that it is workable, its chances of adoption are increased. 

Another Yactor is that if the new idea, practice, or product violates existing value 
structures, it will be harder to adopt. Any innovation that is too radical a depar- 
ture llrom existing value patterns is also in lor tough sledding. Consistent with this 
finding, the innovation proposed should be as congruint as possible with existing 
rvalue structures. i 
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V an^mnovation costs too much money to utilize, its chance* of adoption are 

limited, if the innovations Can be instituted with tittle or no increase in expendi- 
tures, their -eWnces of being adopted are increased. 
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If partial adoptio^i is possible, the Mew idea, practice, or product will more readily 
be utilized. If one can set up a sememe whereby an innovation is adopted on a 
partial basis rather tijan initiaUj/ll-scale integration, its chances of survival are 
increased. 

The strategy for implementing a program evaluation unit will vary from agency 
to agency, due to differences in administrative policy, procedure, etc. Regardless 
of the strategy used, the organizational factors and research regarding change 
proposed here will be helpful\o the agency director in his efforts to get an evalua- 
tion unit implemented. ^ 

Assuming the existence of a program evaluation unit in an agency's operations, 
the crucial issue of assuring implementation ot its findings must be faced by the 
top State administrator. 
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APPENDIX A 



PURPOSE AND CHARGES 

* IRS Study Group I 

on 

Guidelines for Evaluating VocationafcRehabilitation 
Programs and Services 

Purpose . 

Tlje purpose of the study group is to develop criteria and methodolopy for evalua- 
ting effectiveness arufquality of services within State vocational rehabilitation 
programs. 

CHARGES 



1 . Explore currerffstate program evaluation uifrthodsvp.g., 

North Carolina, Florida, Michigan, Texas, Wisconsin; review (<~y 
other pertinent social agencies and evaluative methods; and 
review and analyze various methods (PPBS, OPS, MBO, 
PERT.CPM). 

2?.* Develop guidelines for evaluating State agency programs in 
such ureas as: 

(a) Nature and scope of services, including 

- quality of services 

- efficiency ol case management 

- time factor in processing cases 

- attention to* target group, etc. 

(b) Composition, role, and functions of a model Program 
Evaluation Staff 

(c) Capacity to provide planned services 

- utilizing of present staff 

- staff development 

(d) Public relations and public information (how the 
State agency perceives and publicizes itself) 
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(e) Effectiveness of use of financial resources ~ 

(f) Effectiveness of working relationships with public 
and private agencies and facilities 

Give recommendations! for, implementation and use of guide- 

* 

lines 

(a) Give citations of effective instruments in use 

(b) Coordinate with CSAVR Committee on goals and 
standards, statistics, and the Ad iioc Committee on 



Evaluation. 
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APPENDIX C 



MEMBERS OF TOTAL STUDY GROUP 



Adams A. C Assistant Director of 

Rehabilitative Services 
Departnwnt of Institutions 
Social and Rehabilitative Services 
P.O. Box 25352 

Oklahoma City, Oklahoma 73125 

Alonso, Gerald, Program Supervisor 
Division of Vocational Rehabilitation 
725 South Bronough Street 
Tallahassee, Florida 32304 

Anderson, Earl H.", Assistant Director 
Vocational Rehabilitation Division 
79 Mam Street 
Moll tpelier, Vermont 05602 

Anmow, George 

Seminal Supervisor if Education 
Massachusetts Com/ission for the Blind 
39 Boylston Street I 
Boston, Massachusetts 02116 

0 Athon,Troy^ Chiei Coordinator 
Rehabilitation Standards and Medical 
Services 

Office of Rehabilitation Servkes 
629 State Office Building 
Atlanta, Georgia 30334 

Baptista, Joseph, Chairman 
District Supervisor 

New Jersey Rehabilitation Commission 
150 East State Street 
Trenton, New Jersey 08625 

Barillas, M. G., Assistant Director 
Institutional Hoinebouhd Service Unit 
Rehabilitation Education and Services 
801 Bankers Trust Building 
Des Moines, Iowa 50309 



Ha*Stflt,Fcul, Diiectoi 
Resewcl) and Staff Development 
Department of Vocational Rehabilitation. 
4615 We>t Bioad Street 
P.O.Box 11045 
Ridiinoud.VirtiiuL '23230 

Butenian, Rodney. Supervisory Auditor 
l>iviwon ol Stalt and Load Audits 
600 Sixth, B.W. 
W«bhini:ion,D.C. 20^01 

Bcradcy, William, Program Planning 

F.vyluation and J\e$sajc)) Consultant 
Vocational Rehabilitation Services 
2129 Etist South Boulevard 
Montgomery, Albania 361 H 

Bennett, Carol, Resoura Analyst 
Research Utilization Laboratory 
institute for tht Crippled and Disabled 
340 Eust24tb Street 
N ew Y oft , New Yor k 1 00 1 0 

V 

Blankeuship, Les. Pfognun Analyst 
Evaluation and Mon it orhig Branch 
Rehabilitation Services Administration 
330USUwt,(>.W. 



Wadii&ton, I). C. 20201 

Ble6sot r Trvy A., Atsislwit Regional 

Repjesent alive 
Rehabilitation Services ; 
50 Seventh Street, N.K., Room 44U 
Atlanta. Georgia 30323 

Bonillu, Luis A. 
Bok)U1 

Department of Social Service? 
Vocational Rehabilitation 
HatoRcv. Puerto Rico 1)0919 
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Breeding, Paul A., Director 
Program Planning 

Department of Vocational Rehabilitation 
4615 West Broad Street 
P.O.Box 1 J 045 
Richmond, Virginia 2S230 

Brinson, Leslie C. 

Ducctoj for Research Consultation 
Diiiskm of Voculiona] Rehabilitation 
Department of Public Instruction * 
Rabigh, North Ca olina 27602 

CaranO, Joseph A. 
Acting Training Direcloi 
Bureau of Community unci Institutional 
Servkvs 

Division of .Vocational Rehabilitation 
61*0 Asyhui) Avenue 
lWtfoni, Connecticut 061 OS 



CMjte, James H. 

ASs&tanl Regional Representative 
Rehabilitation Services Administration 
• P. 0. Box 12<>00 . 

Philadelphia > Pemn y Jvania 1 9 1 08 

i 

^jto)c, Le&> Acting Chief 
^E\aluation and Monitoring Branch 

Rehabilitation Services Administration 

Washington, D. C\ 20201 



Coryell, D. Wayne, Program Supervisor 
Division of Vocational Rehabilitation 
725 South Dronough Street 
Tallahassee, Florida 32304 

Covington, George, Assistant Deputy 
Director 

Advisory Services and Special Programs 
Division of Vocational Rehabilitation 
227 South Seventh Street 
Springtjrtd, Illinois 62706 

Ducnwald. Lloyd « " 
Service to the Visbally Impaired 
222 EaLt Cupitol Avenue 
Pierre, South Dakota' 57501 

Duseubury, J. S., Assistmxt Commissioner 
Field and Case Services 
Vocational Rehabilitation Department 
400 Wade Hampton State Office Building^ 
Columbia, South Carolina 29201 

Evunkojohn, District Supervisor 
Office of Vocational^Rehabilitation 
125 Main Street 
Buffalo, New York 14203 



Collignon, Fred 

Department of City and Regional Planning 
University of California at Berkeley 
Beikctey. California 94720 

Collins. John A., Coordinator 
Service Standaids 

Division oJ Vocational Rehabilitation 

623 East Adams Stieet 

Springfield, lUinois 62 7 06 « 

Cook, Call 

Office of Sei vices ior the Blind 
'Department ol Social Services 
300 South Capitol Avenue 
Commeice Centei Building 
Unsing> Michigan 48926 



Feyfell, William J., Rehabilitation and 

Home Teacliing Specialist 
Services foi the Blind 
303 State Oifice Building 
Nashville, Tennessee 37219 

Giovannvni, John P. 

Pnjyect Admbiistraior 

Program Planning and Evaluation 

Systems Project 
Divisiorhof Vocational Rehabilitation 
1 West WHaaHStreel, Room 720 
Madison, Wisconsin 53702 

Humlett Carter D. 

Rehabilitation Counselor 

Virginia Commission foj the Visually 

Handicapped 
124 Church Avenue, S.W. 
Roanoke, Virginia 24001 
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Hammond, N. Spencer Leary, James 

Administrative Assistant Division of Vocational Rehabilitation 

Vocational Rehabilitation Services Division Administrative Office 
Department of Social and Health Services Box 1 037 
P.O.Box 1788 
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Olympia, Wasliington 98504 

Hazian, Decpn J., Supervisor 
Rhode Island division of Vocational 

Rehabilitation 
40 Fountain Street 
Piovidence, Rhode island 02003 

,Hoose, Erwin 

Supervisor oPCasework Operations 
Eastern Regional Office 
Suite 305 , Metropolitan Building 
Hint, Michigan 48502 

Hope;Rob/rt C M Supervisor 
Program Planning and Evaluation 
* Rehabilitation Services 
21 I'Broadway, Room 227 
Little Rock, Arkansas 72201 

Kappes, Donald 

Ptafnning and Evaluation Specialist 
♦Vocational Rehabilitation Service 
P.b. Box 1190 
Wilmington, Delaware 19899 

O'Keefe, James 

Assistant Director Case Services 
State Services for the Blind and 

Visually Handicapped * */ 

1745 University Avenue 
St, Paul, Minnesota 55404 m 

Keith, William, Director # • 
Section of Vocational Rehabilitation 
State Department 6f Education 
* Farm Bureau Building 
1 61 6 Missouri Boulevard • 
Jefferson City, Missouri 65101 

Kolber, Philip, Training Coordinator 
Division of Vocational Rehabilitation 
623 East Adams Street „ 
Springfield, Illinois 62706 



Bismarck, North Dakota 585^1 t 

Leary, Paul A., Director of Training 
Rehabilitation Research and Training 
Center 

University of West Virginia 
Institute, West Virginia. 25112 

Lee, Frank 

State Vocational Rehabilitation 
1808 W. Enai Road, Room '1 400 
Nashville, Tennessee 37203 

Lillie, Ken 

Division of Vocational Rehabilitation 
6)5 State Office Building 
Atlanta, Georgia 30334 

McHugh, Edward 

Assistant Director of Training 

Massachusetts Rehabilitation Commission 

296 Boylston Street 

Boston, Massachusetts 021 1 6 

Marinelliu Adriano 
Rehabilitation Commission 
Labor and Industry Building 
' Trenton, New Jersey 08625 

Martin, Ctytie A. > Supervisor 
Staff Development 
. Rehabilitative Services 
P.O. Box 25352 

Oklahoma City, Oklahoma 73125 

Meeksi, George E., Assistant Director 
Cooperative School Programs 
Department of Vocational Rehabilitation 
4615 tVest- Broad Street 
P.O.Box 11045 
Richmond, Virginia 23230 
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Nflchael, Thomas C, State Supervisor 
Virginia Commission for the Visually 

Handicapped 
Vocational Rehabilitation Departmem 
3003 Parkwood Avenue 
"Richmond, Virginia 23221 

Morehead, wflF., Assistant Director 
V Progrwy Evaluation 
VTexas Rehabilitation Commission 

Medical Park Tower 
•1301 West 38th Street 

Austin, Texas 78705 

Moriarty, Joseph B., Director 
j(ehabilitation Research ^nd 

Training Center 
University of West Virginia 
institute, West Virginia 25 11 2 

Nitardy, Karl, Psychologist^ 
State Services for the Blind and 

Visually Handicapped 
Department of Public Welfare 
1745 University Avenue 
St. Paul, Minnesota 55104 



Otttnar, Chris, Rehabilitation Specialist 
for Planning and Special Programs 
Division of Vocational Rehabilitation 
1 745 University Avenue 
St. Paul, Minnesota 55104 

Pacioeco, Justin T., Assistant Director 
.Rehabilitation Service Operations 
Impertinent of Vocational Rehabilitation 
4615 Wes: Broad Street 
P.O. Box 1)045 
Richmond, ^irgiuia 23230 

Prouty, Robert H. 

Coordinator of Administrative Services 
Section of Vocational Rehabilitation 
State Department of Education 
Farm Bureau Building 
1616 Missouri Boulevard 
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GLOSSARY 



ACCOUNT CLASS - A descriptive heading or numeric code used to categorize 
similar financial transactions according to program, function, object, or 
source, contained in a chart of accounts. 

ACCRUAL ACCOUNTING - An accounting system in which revenues and ex- 
penditures are recognized as they are earned or occur regardless of when pay- 
, ment is made or when the income is actually received. B This system reflects 
/ the resources available to an agency, the receipt of goods and services, the 
/ use of resources irr relation to work performed and benefits derived during a 
particular time period, and the liabilities of the agency. For management it 
enables moie effective controls because it provides data on all available 
resources and on expenses that can be compared with and related to program 
performance during a given period. Accrual accounting in Federal agencies 
inquired by P.L. 863 (August 1 , 1956), Frequently it is contrasted with- 
the cash basis of accounting which emphasizes cash receipts and disbursements 
during a given period. * 

ALTERNATIVES - Within any one agency, this term means other possible pro- 
grams besides those already decided upon. It denotes a comparison of two 
or mom programs (that is, two or more possible approaches) as possible ways 
of fulfilling the same objective. Used in this context the term is output- 
oriented, it suggests substituting an entirely different program (and therefore 
a different output or outputs) for a program already planned or in process. 
On the other hand, alternative ways to do a job which has been decided 
upon takes the program as given, and raises possibilities for changing the mix 
of inputs." 

ANNUAL PLANNING CALENDAR - Phases of current organizational activities 
are integrated with long-range planning on scheduled annual cycles to coordi- 
oate procedures for all agencies or program centers; time schedules may be 
designed . 

APPROPRIATION - An allocation of funds made by a governing authority for 
specified purposes and often restricted as to the time when it may be 
expended. 

BENEFIT-COST ANALYSIS • • (Sec Cast-Benefit Analysis) 
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BUDGET - A financial plan serving as a pattern Jot and control over Mure opera- 
tions, hence, any estimate future costs any systematic plan for the utilisation 
of manpower, materia), or other resources. The term '•budget** in tlie Federal 
budget context also refer* to the summary totals oi appropriation, receipts, 
expenditures (excluding net )endingj ? expenditure account suq>lus or deficit. 

BUDGETING - The process of translating planning and programming decisions 
into specific projected financial plans for relatively short periods of time. 
Budgets arc short-range segments of adopted action programs which set out * 
planned accomplishments and estimate the resources to be applied for the 
budget periods in order to attain those accomplishments. 

BUDGET DOCUMENT A written statement of an estimate or plan desaihing 
expenditujes and revenues for financing an organization's entire program for 
a specified time period, usually one year, the most common fiscal year be- 
gins July ) and ends Jane 30. at the termination of the defined period , Iftfe- , 
budget technically no longer exists, other than as a historical document. * k 

BUDGETARY PROCESS - A continuous activity comprised of planning, formu- 
lation of a budget document, interpretation, presentation to the approving 
authority, formal adoption, fiscal administration, and anpraiyal. 

COMPONENT • Level of program subordinate to element level and above task 
level. 

COS T-BASED BUDGETS Budgets in which activityjevels are to be estimated 
in terms of value of resources to be consumed, in carry mg out the activity, 
rather than in terms of obligations incurred or payments made. These re- 
source requirements, when distributed to program elements and categories, 
piovidc a cost basis for program planning>and budgeting. * 

COST-BEnCfIT ANALYSIS A means of assessing the worth of existing and 
proposed projects, it involves the enumeration and evaluation of all relevant 
costs and benefits over a period of time, ideajjy, benefits should exceed 
costs. 01 £> 1 . measurement criteria for the benefits should be specified. 
Tins is an analytical approach to solving problems of choice. It requires the 
definition of objectives, identification of alternative ways of achieving each 
objective, and the identification, for each Objective, of that alternative 
which yields the greatest benefit foj a given cost or that alternative which m 
produces the required level of benefits at the lowest cost litis same analyti- 
cal pioccss has also been referred to as ^ost effectiveness analyse whtflVthc 
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COST-BENEFIT ANALYSIS - (Contd .) 

benefits or outputs of the alternatives carmot be quantified in terms of dollars. 
However .there islncfeasing interest in.combining non-economic benefits with 
dollar benefits in evaluating particular programs, and methods for doing this 
are ^eing developed . 

COST-EFFECTIVENESS ANALYSIS - A means of relating the cost of a particular 
activity or project to effective performance or goal attainment: the decision 
maker may choose from among feasible alternatives on a basis of least cost 
and greatest effectiveness. 

CRITERIA - Measurements which are used to examine the relathe degrees of 
desirability among alternatives or the degree to which a course of action 
meets an intended objective. 

CRITICAL PATH METHOD (CPM}'~- Network Analysis model. It has its own 
modeling language; it differs from PERT in only one fundamental respect: 
CPM seeks to determine the expected limes of completion of the total pro- 
ject and times of completion of the subprojects of which it is composed. 
PERT goes further and seeks to estimate' variances associated with these 
expected times of completion. ' , 

CROSSWA& - The expression of the relationship between the program struct 
ture and the appropriation -budget structure; the translation of multiyear 
program and financial plans into annual budgets: a simple table vertically 
listingprogram categories and horizontally listing appropriations and budget 
activities, based upon the program budget code. % 

DIRECT COSTS - Actual or budgetary costs that may be charged directly to, 
or prorated as a part of. the cost of a program . service, function, or depart- 
ment. They are eliminated if a program is eliminated or added if a program 
is added. 

EFFECTIVENESS - The performance or output received from an approach or 
a program. Ideally it is a quantitative measure which can be used to evaluate 
the performance level achieved in relation to criteria pertaining to end objec- 
tives. An example of such a measure would be the increase in annual earn: 
ings of a group of participants in a Federal retraining program- This example 
assumes that an objective of the retraining program is to increase tire level 
. , of income of program participants. Under this assumption, a measure of 
output, such as the number of people who completed the program, while 
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EFFECTIVENESS (Contd ) 

infornvitive, would noi lx\a valid measure of effectiveness amce the objective 
is to increase income, not merely to re tram people. 

ELEMENT - Level of program Mibordinate to program level and abo\e compon- 
ent level, cost elements include personal services, contracted services, equip- 
ment, materials, supplies, and fixed charges. 

« 

EVALUATION - Comparison of desired outcomes or objectives with actual 
accomplishments, based upon educational performance indicators, such as 
indices that measure changes in pupil cognitive development. 
« * 

FUNCTION-OBJECT BUDGET WideK used presently by loca)jinbl>c schools 
to identify^costs under a number of bioadly defined funcfioiyand object 
categories, such as administration, instruction, debt service, and plant main- 
tenance; emphasis is upon objects of expense rather thair progiatns of the 
school. — ^ 

GENERAL SYSTEMS THEORY An -interdisciplinary, holistic, integrative 

approach that includes logiobntethodological investigations of the empirical, 
the philosophical basis for systems analysis *nd related procedural strategies. 

HEURISTIC - Solution of a problem by 3. trial and error approach frequently^ 
. involving the act of learning, and often leading to further discovery or con- 
. elusions without providing proof of the correctness of the outcome. 

« 

INDIRECT COSTS - Actual or budgetary costsTftat are not readily identified 
with a specific program, service, function, or department and that are sel- 
dom completely eliminated >f a program is eliminated. 

INPUT-OUTPUT ANALYSIS An economic technique designed td examine ihe 
effect (if changes in certain input variables to the outcome or output variables 
* of the system under study , a foim of n> stems analysis: inputs are the resources 
employed to achieve objectives and outputc arc the products of a program, 
often expressed numerically 

MANAGEMENT BY OBJECTIVES (MBO> - Management approach that tends to 
minimize undesirable behavioral effects. The principal feature of this type 
of management is the establishment of specific performance goals for each 
. position, particularly for each managerial position. By stressing these objec- 
tives, overall control is achieved through self-control by individual participants 
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(MBO) -- (Contd.) 

. Rather than applying control from above, the emphasis is placed upon con- 
trol from within. Of course, establishment of objectives and appraisal of 
performance is performed under the direction of a lugher-level manager. In 
each case, however, the stress is upon accomplishments and results. MBO 
distinction it found in its careful delineation of formal objectives for a 
specific time period. 

MANAGEMENT INFORMATION SYSTEM (MIS) - Integrates the dynamic func- 
tions of an organization, such as instruction, personnel and finance, and pro- 
vides computer-aided systems of information control for administrators; it 
may be a reporting system or a decision-making system, depending on level 
of application. 

MARGINAL ANALYSIS - The process of identifying thebenefitt. or costs of 
alternative behaviors as unitary changes in the alternative variables occur and 
equalizing the benefit-cobt ratio to form a point of indifference (tradje-off) in 
benefit per additional unit of resources input for dccisionnnaking purposes. 

MULTIYEAR PROGRAM SUMMARY - Concise description in numerical, dollar, 
and other values forthe past year, current year, void future year projections, 
the authorized program output, input, personnel and material needs, and 
proposed program changes. 

NEEDS RESEARCH - Formalized studieb and problem-solving steps designed to 
specify the most appropriate level of systems -analysis to be used. 

NETWORK ANALYSIS - A technique used in the planning, scheduling, and 

solving of problems related to large-scale projects which involve a great num- 
ber of interrelated decision' points or events. The project is displayed as a 
network which connects these points or events in such a way as to show the 
various alternative "paths". which may be taken from each point. When used 
in scheduling, each event is dependent upon, certain necessary events having 
preceded it. These dependencies are portrayed by the connecting aspects of 
the network. 

. OPERATIONS ANALYSIS - A term used by the US Office of Education to des- 
cribe quantitative, analytical studies in education; it combines operations 
research and systems analysis. 
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OPERATIONS RESEARCH (OR) - The use of analytic methods adopted from & j 
mathematics and other disciplines for solving operational problems. Among, 
the common techniques used in operations research are. linear programming, 
probability theory, deasion theory, Monte Carlo methods, and queing tech- 
niques. ^ 

PERFORMANCE BUDGET - Used to evaluate work-cost 4ata in terms of unit 
work measures, the forerunner of the program budget; led to the introduc- 
tion of activity classifications, evolved m the era of scientific management ; 
provided numerous workload statistics but did not aid greatly in planning 
future courses of action . 

PLANNING Planning in the long^ange sense is the selection or identification'of 
the overall, long-range objectives of the organization a\td the analysis of 
various possible courses of action in terms of their relative costs and accomp- 
lishment^ ber/efits in order to decide on which courses of action (such as 
program^ to follow in order to achieve those objectives. The analyses re- 
quired are variously deferred to as cost<0ectivenesi>. cost utility , or cost- 
benefit (benefit-cost) studies. Essentially, this type of planning .involves 
deciding on what the organization is m business to do and generally how it 
is to be done. This is also called strategic planning. 

c 

PROGRAM BUDGET - Relates resources, financial and otherwise, to an organi- 
sation's 3ctMties, outputs, services, missions, or programs; the finar/cial 
expression o^ value priorities, helps to achieve cost-effectiveness if not cost- 
reduction, based upon a program structure classification: the budget is a 
statement of policy that relates cost to differential programs; sometimes 
used in a broad sense to denote the entire process of PPBS. 

PROGRAM CATEGORY - A.dSDssification within a program structure which 
groups progran* which have the same or similar objectives. 

PRpdR&M ELEMENT - A subdivision of a program category which comprises 
( the specific products that contribute to an agency's objective(s). If an 
agency's operating program is distributed over several program categories, 
each part of the operating program identified by a discrete program category 
is a program element. 

PROGRAM EVALUATION AND REVIEW TECHNIQUE (PERT) - PERT and 
CPM (Critical Path Method) are network analysis models N Each has its own 
modeling language, but they differ in only one fundamental respect : CPM 

V 
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(PERT)-(Contd.) 

seeks to determine the expected times of completion of the total Rroject 
and times of completion of fbe subprojecti of which it js composed. PERT 
goes further and seeks to cstitnatc variances associated with these expected 
tunes of completion. 

PROGRAM MEMORANDUM - An internal planning docunrent that records 
analyzed programs and lists alternatives and recommendations. 

PROGRAMMING - Programming is the process of deciding on specific courses 
of action to be followed in carrying out planning decisions on objectives. It 
also involves decisions in term3 of total costs to be incurred over a period of 
years as to personnel . material, and financial resources to be applied in carry- 
ing out programs. ^ / 

PkOGRAM STATE\?ENT - A forma), recorded description of multiyear pro- 
gram needs, objectives, authority , inputs, outputs, and supportire informa- 
tion. 

PROGRAM STRUCTURE - Organization of the genera) program areas, subpro- 
grams, elements, components, and tasks; it facilitates analytic comparisons 
of the costs and effectiveness of alternative programs; programs may cut 
across existing departments and agencies. 

PROGRAM SUBCATEGORY - A subdivision of a pro-am category It combines 
agency programs or activities on the basis of narrow objectives within the 
broader objectives of the program category. 

PRORATION OF COSTS - The distribution of costs to two or more progi-am 
areas in proportion to the benefi/fs provided ; basis for proration may be a 
formula or some ather arhixrarilyWtermined procedure. 

QUANTIFICATION ^ The numerical expres^on of variables. 

SCENARIO ~ A statement of assumptions about the operating environment of 
the organization to be studied: it is a helpful aid m making projections of 
different future conditions. 

-STATISTIC - A measure , quantity or value which h calculated from a sample 
rather than from the population. V ^ , 

STATISTICAL INFERENCE - Using information contained in a sample to make 
• predictions about a larger set, the population. 
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SYNERGISM - Coopeiative action of discrete .units or agencies which results 
in a total effect that is greatej than the sum o* those effects taken independ- 
ently. J 

SYSTEMS ANAJ. YSIS - Systems analyst may be viewed as the search for and 
evaluation of alternatives which are relevant to defined objectives, based on 
judgment and, vUerever useful on quantitative methods, with the objective 
of presenting such evahuttoits to decisionmakers for their (onsidciation. If 
emphasizes the system concept under which any course^of action designed 
to achieve an objective is viewed as a system requiring inputs and producing 
outputs. The inputs and outputs involved may take on any of a large variety 
af forms. In this sense, system analysis cncompaeses-Uoth cost-benefit and 
oost-effectiveness analyses, as well as other types of analysis which may be 
more limited in scope. / 

1 

UNASS1GNED SUPPORT - A budget category containing items that are not 
assigned to more direct program, service, or function -object categories; a , - - 
kind of residual budgetary item. 

VARIABLE-- - A quantity that may increase or decrease without other essential 
changes. 

WELFARE ECONOMICS - The study of the economic weJj-being of all persons 
as consumers anchts-producers, and possible ways in which that well-being 
maybein/proVsd. JUs also known as NORMATIVE PRICE THEORY. 



A BRIEF BIBLIOGRAPHY ON PROGRAM EVALUATION 



The first group that follows consists of items for quickly getting an overview of 
issues, and occasionally methodology problems, relating to evaluation of social 
programs generally. The second group consists of selected key s^udfcs or over- 
view essays relating specifically to rehabilitation programs. 



GENERAL 

American Institutes for Research, Evaluative Research: Strategies and Methods , 
(Pittsburgh: AIR, 1970) - papers and summary of discussion at conference, 
good reading. Note especially the paper by David Hawkridge, "Designs for 
Evaluative Studies." 

Caro, Francis G., "Approaches to Evaluative Research: A Review," in Louis Zurcher 
and Charles'Bonjean (eds.), Planned Social Intervention: An Interdisciplinary 

' Anthology , (Scranton: Chandler, 1970). pp. 403-42 1 ; also in Human Organi- 
zation, 28 (Summer, 1969), 87-99. 

Caro, Francis G., ed., Readings in Evaluation Research , (N.Y.: Russell Sage Found- 
ation, 1971) - a very fine collection of readings, including examples of applied 
evaluation research. Comparable in quality to the Weiss reader with more 
emphasis on methodology, but available only in hardcover. Contains the 
Caro summary article. j** 

Evans, John W., "Evaluating Social Action Programs," in Zurcher & Bonjean; also 
in Social Science Quarterly, 50 (December, 1969). 

Glennan, Thomas K., "Evaluating Federal Manpower Programs: Notes and Obser- 
vations," RAND Corporation, Memorandum RM-5743-OEO (September, 
. 1969) -also in Weiss. 

Rein, Martin, "Social Policy Analysis as the Interpretation of Beliefs," Journal of 
the American Institute of Planners; (September, 1971), 297-310. 

Rossi, Peter H., "Practice, Method, and Theory in Evaluating Social Action Pro- 
grams," in James L. Sundquist, ed., On Fighting Poverty: Perspectives from 
Experience, (N.Y.: Basic Books, 1969), pp. 217-234, 

Suchman, Edward. Evaluative Research ( N. Y.: Russell Sage Foundation, 1967), 
minor "classic" of field, mbSt often used text. 

m 

Tri^odi, Toly, Social Program Evaluation: Guidelines for Health, Education and 
Welfare Administrators , (Itasca, Illinois: F.B./Peacock Pub., 1971)-ashort 
and simple book with got>d overview of methods, fine introduction. 

f 
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Weiss, Carol, cd., Evaluating Action Programs, (Boston: Allyn & Bacon, 1972) - 
your BEST BUY, a truly superb reader, uniformly excellent, contains shorter 
articles by Suchman, Rossi, Wholey et al, Evans, etc., as well as many fine 
articles and papers not available elsewhere. 

Wholey, Joseph S., et al, Federal Evaluation Policy: Analyzing the Effect of 

Public Programs (Washington, D. C: The Urban Institute, 1970) - assessment 
of evaluation activities of OEO, HEW, HUD, LABOR. 
<s> /" ' 

WilliamsvWalter, Social Policy Research and Analysis: The Experience in the 
Federal Social Agencies (N. Y.: Elsevier, 1971) - similar to Urban Institute 
Stuo\, with more examples, aimed at teaching evaluation. 

SPECIFIC TO REHABILITATION 

Biscamp, Larry, Charles Cole, Judy Taylor and Herbert Willsmore, "A Client 
. Evaluation of Rehabilitation Counselor Training Programs," Institute of 
. Urban and Regional Development, University of California, Berkeley, 
Working Paper No. 6. 

Cohen, Julius, Irene Butter, Stanley Deline, and Ronald Nutter, eds., Benefit- 
Cost Analysis for Mental Retardation Programs: Theoretical Considerations 
and a Model for Application (Ann Arbor: Institute for the Study of Mental 
Retardation and Related Disabilities, University of Michigan, 1971). 

-CoUignon, Frederick, Adam Zawada, Barbara Thompson, and Joel Markowitz, 
"Guidelines and Criteria for Evaluating Vocational Rehabilitation Programs" 
Institute of Urban and Regional Development, University of California, 
Berkeley, Working Paper No. 3. 

CoTiley, Ronald W., "A Benefit-Cost Analysis of trie Vocational Rehabilitation „ 
Program," Journal of Human Resources , IV, No. 2 (Spring, 1969), pp. 226-252. 

"Conley, Ronald W., The Economics of Mental Retardation, (Baltimore: Johns Hop- 
kinsTress, 1972). 

Conley, Ronald W., The Economics of Vocational Rehabilitation , (BaltiaKffe: 
Johns Hopkins Press, 1965). 

Grigg, Charles, Alphonse Holtman, and Patricia Martin, Vocational Rehabilitation 
of Disabled Public Assistance Clients: An Evaluation of Fourteen Research * 
and Demonstration Projects, Institute for Social Research, No. 8, Tallahassee, 
Florida: The Florida State University, 1969). 

r 

Heferin, Elizabeth A., and Alfred H. Katz, "Issues and Orientations in the Evalua-' 
tion of Rehabilitation Programs: A Review Article," Rehabilitation Literature , 
Vol. 32, Part I (March, 1971), pp. 66-73, and Part II (April, 1971);pp. 98-106. 
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Michigan Department^ Education, Division of,VocationaTRehabilitation, "The 
Vocational Status of Michigan Rehabilitantfof Fi seal Year 1969 Two Years 
After Case Closure, The Results of a Follow-Up Study and Benefit/Cost 
Analysis Conducted by the Program Analysis, Planning, and Development 
Section of the Division of Vocational Rehabilitation, " Nlichlgan Department 
of Education, Division of Vocational Rehabilitation, February, 1971. 

Nagi, Saad Z., Disability and Rehabilitation. Iftgal, Clinical and Self-Concepts and 
Measurement . (Ohio State University .Press, 1969). 

Schon, Donald, "The Blindness System." the Public Interest* 18 (Winter, 1971) 
pp. 25-38. . ' * v 

f •• 

Sussman, Marvin B., ed., Sociology and Rehabilitati6n .(American Sociological 
' Association, "l 9 66). Note in particular the essay by Edward Suchman, "A • 
Model for Research and Evaluation on Rehabilitation." 

♦ ** 
.Sutherland, Pat F., "Program Eyaluation in Social Action Programs, with .Glossary," 

' State of Texas Commission for the Blind ,1971.* 

Wright, George N., Kenneth W. Reagles,and Alfred J. Butler. The Wood County 
Project, An Expanded Program of Vocational Rehabilitation , (Madison, Wis- 
. consim The University of Wisconsin Rehabilitation Research Institute, 1969). 

ALTERNATIVE APPROACHES TO POLICY ANALYSIS 
AND PROGRAM EVALUATION 

Policy analysis may be approached from a variety of disciplinary and operational 
perspectives. The following references provide an introduction to several bases for 
such analysis. 

Alternative Perspectives " 

Archibald, K. A., "Alternative Orientations to Social Science. Utilization," . 

• Social Science Information , 9 (April 1970). ' \ 

* Archibald, K. A.,'"Three Views of the Expert's Role in Policymaking: Systems 

Analysis, Incrementalism, and the Clinical Approach," Policy Sciences^ 
1 (19.70), 73-86* 

Dror, Yehezkel, "Policy Analysis: A New Professional Role in Government," . 
Public Administration Review. 27 (1967), 197-203.* . • .. 



Piven, Frarfces Fox, "Whom does the Advocate Planner Serve," "Comment" ty 
Sumner Rosen, aft "Reply" by Piven, Social Policy , (May-June 1970), 32-37. 
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Rein, Martin, 44 Social Planning: The Search for Legitimacy " Journal of the 
American Institute of Planners, 35 (1969), pp, 233?244. 

Rein, Martin, "Social Policy Analysis as the Interpretation of Beliefs/' Journal 
of the American Institute of Planners, 37 (1971), pp. 297-310. 



No single book or collection of readings can cover all methods of program planning. 
In part, this is due to the sheer variety of techniques that may be employed in plan- 
ning. But the gap is also due to the chronic division between the teaching of the 
content and philosophy of planning and the teaching of methods. The following 
books may be helpful for thinking about kinds of methodological approaches: 

Rivlin. Alice M., Systematic Thinking for Social Action. Washington, D. C: 
<The Brookings Institution, 1971 . A good overview of analysis for planning 
from the Federal agency point of view. Very suggestive for thinking about 
what kinds of methods might be useful. 

Williams, Walter, Social Policy Research and Analysis. New York: Elsevier, 1971. 
An attempt to look at how Federal social agencies used socjftl research in the 
1960's. Tends to use education examples. / 



Frameworks for Program Planning: PPBS 

The most recent effort to develop a comprehensive approach to program planning. ^ 



Joint Economic Committee, U. S. Congress. The Analysis and Evaluation of Public 
Expenditures; The PPB System. Washington, D. C: U. S. Government Print- 
ing Office-, 3 volumes, 1969. • A colossal compilation of testimony and articles 
(e.g., Fred Hayes on the PPB strategy of .the Lindsay Administration; Wfldavsky 
on "Rescuing Policy Analysis from PPBS) that amounts to 1241 pages. Many 
of the papers were reprinted in Robert H. Haveman and Julius Margolis, 
Public Expenditures and Policy Analysis , Chicago; Markham, 1 970. 

Novick, David, (ed.), Program Budgeting. Cambridge: Harvard University Press, 
1965. The early conventional wisdom. While still being written, this book 
was first published by the Government Printing Office in order to provide 
guidance to agencies during the firtt year pf expansion of PPB to the civilian 
"branches of the Federal Government. 



Schultzc, Charles C The Politics and Economics of Public Spending . Washington, 
D. C: The Brookings Institution, 1968. Schultze Was the Director of the 
Bureau bf the Budget during the Johnson Administration: This book provides 
an economist's political view of the -process of introducing analysis into govern* 
r . ment decisions. 
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Methods for Analysis and Planning 

Basic Statistics: There are innumerable texts, new and old, that can be used to 
refresh your fading memories. What they do not do is tell you how to know what 
numbers you should be looking for. Nevertheless, the great preponderence of pro- 
gram analyses require little more than an adequate background in basic descrip- 
tive statistics, regression, and methods of hypothesis testing, together with some 
knowledge of experimental design and survey methods. Of the current introduc- 
tory texts, the following are recommended: 

Blalock, Hubert M., Social Statistics , New York: McGraw-Hill, 1960. 

Davis, James A.. Elementary Survey Analysis , Englewood Cliffs, N. J.: Prentice- 
Hall, 1971. 

Systems Analysis and Other Applied Quantitative Methods: Operations researchers 
have developed a style of attacking problems and a body of analytical techniques 
" that are very powerful for certain classes of problems, especially those involving 
optimization. Texts are legion. The following is a very good example: 

Ackoff, Russell L., and Maurice W. Sasieni, Fundamentals of Operations Research, 
New York: John Wiley, 1968. 

Systems analysis is usually differentiated from operations research by the scale and 
. fuzziness of the problems it tries to deal with. The following books reflect its multi- 
farious origin in general systems theory, engineering of large physical systems, *nd 
economics and the problems of choice. 

Churchman, C. West. The Systems Approach , New York: Delta, 1968. A non- 
technical introduction. 

Hare, Van Court, Jr., Systems Analysis : A Diagnostic Approach, New York: 
Harcourt, Brace and World, 1967. Attempts to provide a technical overview 
based on general systems ideas. 

*\ 

de Neufville, Richard and Joseph H. Stafford, Systems Analy sis for Engineers and 
Managers , New York: McGraw-Hill, 1971. A good example of the ^lending of 
approaches from economics, operations research and engineering into a text. 

Quade, E. S., and W. I. Boucher (eds.), Systems Analysikand Policy -Planning; Appli- 
cations in Defense , New York: Elsevier, 1968. Systems analysis 
viewfed as a method of problem solving. Examples are defense oriented. 
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Social Research Methods 

Another major group of methodological approaches, heavily focus on survey meth- 
ods and interpretation. The following provide introductions: 

Burton, T. L. and G. E. Cheny, Social Research Techniques for Planners. London: 
George Allen and Unwin, 1970. Not yet in our library. I have not seen it, but 
the title is just right. 

Hauser, Philip M. (ed.), Handbook for Social Research in Urban Areas. Paris, 
UNESCO, 1965. 

Clinical and Behavioral Science Approaches to Planning 
A broad area of great importance to program planning. 



Bennis, Warren G., "Theory and Method in Applying Behavioral Science to Planned 
Organizational Change," in J. R. Lawrence (ed.), Operational Research and 
The Social Sciences, London: Tavistock, 1966. A really fine compact paper 
that covers the subject. 
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